⏳ 1 Day to Go! Flink Forward Asia 2026 — Main Forum Agenda Revealed! FFA2026 brings together world-class tech leaders to showcase the full picture of real-time data intelligence — from Alibaba Cloud's Agent-native to AI-native evolution, and from automotive to embodied AI industry scenarios. 📅 June 26–27 | InterContinental Shenzhen OCT, Shenzhen ⚡ Limited seats available — don't miss out! 📝 Note: All sessions will be conducted in Mandarin Chinese. #FlinkForwardAsia# #FFA2026# #ApacheFlink# #RealTimeData# #DataAI# #AIAgent# #AgenticAI# #StreamingData# #OpenSource# #AlibabaCloud# #EmbodiedAI# #DataEngineering#

# Learning Palantir Foundry 🚀 Are you recomputing billion-row tables in full every single day? Process only the delta, and your compute costs drop dramatically. 📌 Title and Feature URL Title: Incremental Transforms URL: 📝 Overview Incremental transforms enable efficient data processing by handling only the data added or changed since the last run, instead of reprocessing the entire dataset. They're enabled with the `@incremental()` decorator, which automatically chooses between incremental and snapshot execution based on how the inputs changed. 🔧 How It Works The `@incremental()` decorator wraps a transform function to give it delta-processing capability. - It converts the standard input/output objects into incremental variants: `IncrementalTransformInput`, `IncrementalTransformOutput`, and `IncrementalTransformContext` - Input read modes can be `added` (new rows since last run, the default), `previous` (state from the last run), or `current` (the full current dataset) - Output write modes are `modify` (append to existing output) or `replace` (overwrite entirely); the default is `modify` for incremental runs and `replace` for snapshot runs - Key parameters include `require_incremental` (fail if incremental isn't possible), `semantic_version` (bumping it triggers a snapshot rebuild), `snapshot_inputs` (exempt specific inputs from incremental constraints), and `strict_append` (enforce append-only safety) 🛠 Practical Usage - Add `@incremental()` to large append-heavy log or transaction tables to replace daily full recomputes with delta processing - When you change logic, bump `semantic_version` to safely trigger a snapshot rebuild - Use `require_incremental` to force delta execution when you don't want a silent full reprocess - Use `strict_append` when you need strict append-only guarantees 🎯 Use Cases - Slashing soaring compute costs from daily full recomputes of billion-row tables via delta processing - Serving as the core cost-optimization technique that determines the economics of large-scale projects - Daily ingestion of append-only transaction histories and event logs - Streamlining pipelines whose upstream grows only through additions (APPEND/UPDATE) ⚠️ Caveats - Preview features always run non-incrementally - Unless requirements are met (all non-snapshot inputs contain additions only via APPEND/UPDATE, the input list stays stable, `semantic_version` is unchanged, etc.), the transform automatically runs in snapshot mode and fully replaces the output - Updated or deleted input files must be marked as snapshot inputs - The `previous` mode requires schema validation matching the previous output structure - Transform logic must support both incremental and snapshot execution paths #PalantirFoundry# #DataEngineering#

# Learning Palantir Foundry 🚀 Bring complex logic that no-code can't reach into your data platform, along with full software-engineering quality control. That's what Code Repositories delivers. 📌 Title and Feature URL Title: Code Repositories (Python Transforms) URL: 📝 Overview Code Repositories is a web-based integrated development environment (IDE) for creating and collaborating on production-ready code within Foundry. It provides a friendly UI over the underlying Git repositories, so teams can work without command-line access. With platform-specific features, you can apply software development practices directly to data engineering. 🔧 How It Works Version control and collaboration are at its core. - Common Git tasks (branching, committing, release tagging) execute through the web UI - Pull requests drive code review, with "highly configurable" permissions that support quality assurance such as mandatory reviews - IntelliSense, linting, error checking, and contextual help dialogs are available across all repository types - Transforms repositories let you author data transformation logic in Python, Java, or SQL with preview and debugging - Functions repositories natively integrate the Ontology and run low-latency business logic in TypeScript or Python 🛠 Practical Usage - Use PySpark to implement billion-row entity resolution and complex business rules in code - Require PR reviews so a second reviewer and CI checks must pass before merge - Add unit tests to guard transform logic against regressions - In Functions repositories, leverage Ontology-data-type autocomplete to write logic safely - Bring machine learning workflows into the platform via model development repositories 🎯 Use Cases - Implementing complex reconciliation and business rules in PySpark that Pipeline Builder can't express - Structurally eliminating "regressions from editing production directly" through mandatory reviews and branch-based workflows - Implementing derived KPIs and validation logic as Functions reused across apps - Managing ML model training and inference code under governance ⚠️ Caveats - The docs note that Japanese translations are machine-generated and unverified, so localized content may have accuracy limitations - Each repository type (Transforms/Functions/Model) supports different languages and purposes, so pick the one that fits your goal - Being a pro-code environment, the quality benefits only materialize if your organization establishes review, CI, and test practices #PalantirFoundry# #DataEngineering#

🏗 Migrating tens of thousands of jobs that ingest petabytes a day, without ever stopping data delivery. Meta's playbook of shadow then reverse-shadow then cleanup deprecated the legacy system 100%. Title: Migrating Data Ingestion Systems at Meta Scale URL: 📝 Overview Meta incrementally scrapes several petabytes of social-graph data daily from one of the world's largest MySQL deployments into its data warehouse. This post explains how they migrated tens of thousands of those ingestion jobs to a new self-managed service without disrupting analytics, reporting, and ML pipelines. ❓ Challenges Solved The legacy system was customer-owned pipelines, fine at small scale but unstable at hyperscale. They had to meet increasingly strict data landing-time requirements while migrating without interrupting data delivery across the organization. 💡 Methodology & Proposed Approach They migrate through a three-phase lifecycle. ・Shadow: in pre-production, consume production data while writing to isolated tables, continuously monitoring row-count and checksum mismatches against production jobs ・Reverse shadow: promote shadow jobs to production tables and send the original production jobs to shadow, keep comparing outputs for quality signals, and roll back instantly if needed ・Cleanup: deprecate old jobs after confirming consistency ・Each job is verified on four axes (zero differences, landing latency, resource usage, custom criteria), and CDC maintains full-dump, delta, and target tables 🎯 Use Cases It informs migrating large data-ingestion platforms, phasing CDC pipeline cutovers, and designing zero-downtime system replacements. 📊 Outcomes ・100% of the workload was migrated and the legacy system fully deprecated ・Job status signals streamed continuously to Scuba, and a migration tool monitored each job and auto-promoted/demoted between stages to manage thousands of concurrent migrations ・Bad partitions were flagged in metadata to prevent propagation to downstream jobs and trigger alerts ・To handle capacity limits, they reused old-system snapshot partitions as initial snapshots to cut full-dump load, and the resulting data-quality analysis tool is still used in release validation after the migration #DataEngineering# #DataInfrastructure#

# Snowflake Features and Practical Usage 🚀 "Bumping the size up makes it faster, but what about cost?" Snowflake cost optimization hinges on answering that question correctly. Let's master virtual warehouse sizing and auto-suspend. 📌 Title and Feature URL Title: Working with Virtual Warehouses URL: 📝 Overview A virtual warehouse is a cluster of compute resources that supplies the CPU, memory, and temporary storage needed to run SQL queries and data operations such as INSERT, UPDATE, DELETE, and COPY. It consumes credits only while running and can be resized or auto-suspended flexibly. Designing size and auto-suspend per workload is the first step in Snowflake cost optimization. 🔧 How It Works Key facts about warehouse sizing and billing: ・Sizes range from X-Small to 6X-Large, and each step up doubles compute and credit consumption. X-Small=1, Small=2, Medium=4, Large=8, X-Large=16, 2X-Large=32 ... up to 6X-Large=512 credits/hour. ・Billing is per-second with a 60-second minimum each time a warehouse starts or resumes. For example, an X-Large running 61 seconds costs about 0.271 credits, while a full hour costs 16 credits. ・Larger warehouses speed up large, complex queries, but larger is not necessarily faster for small, basic queries. ・Besides standard warehouses, Snowpark-optimized warehouses target memory-heavy workloads like ML training. 🛠 Practical Usage ・Use AUTO_SUSPEND (on by default) to suspend after idle time and AUTO_RESUME (on by default) to resume when a statement arrives, preventing wasted credits while idle. ・Create with CREATE WAREHOUSE etl_wh WAREHOUSE_SIZE = XLARGE; for batch, and use WAREHOUSE_SIZE = SMALL AUTO_SUSPEND = 60 for ad-hoc analytics to "pay only for what you use." ・Add INITIALLY_SUSPENDED = TRUE to create the warehouse in a suspended state. ・Warehouses can be resized even while running, so you can temporarily scale up just before a heavy job. 🎯 Use Cases ・Run a daily batch on X-Large to finish fast. Since one size step roughly doubles speed and halves runtime, you cut wall-clock time at a comparable credit cost. ・Set an ad-hoc analytics warehouse to Small with AUTO_SUSPEND=60 so it costs nothing when nobody is querying. ・For data loading, small-to-medium sizes are often sufficient; tune based on file count and size rather than warehouse size. ⚠️ Caveats ・Every resume bills a 60-second minimum, so an extremely short AUTO_SUSPEND (a few seconds) can backfire by triggering frequent start/stop cycles. ・A large size is wasted on small queries. "Scale up for slow queries" is the rule — bigger is not universally better. ・Loading performance depends more on file count and size than on warehouse size. Consider parallelizing files before scaling up. #Snowflake# #DataEngineering#

Some data engineering roles end at the dashboard. This one helps stop real-world crime. TRM Labs is hiring a 𝗙𝗼𝗿𝘄𝗮𝗿𝗱 𝗗𝗲𝗽𝗹𝗼𝘆𝗲𝗱 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 to build the systems that power investigations at federal agencies. 🚀 Apply here:

We’re making it easier to develop agents—with new data agents and tools: 1. Conversational Analytics in BigQuery (in preview) 2. Data Engineering Agent (GA) 3. Data Agent Kit (in preview) 4. Managed MCP Servers for Databases (GA) + more →

Hi @X, looking to #connect# with more builders. Into: 🧱 Data Engineering 🔥 PySpark / Spark 🧊 Databricks & Delta Lake ☁️ Azure (ADF, ADLS) 🤖 AI/ML & LLM Pipelines 📊 Vector DBs 🌊 Streaming 🗄️ SQL 🚀 Startups 🌍 Open Source Let's connect, learn in public, build cool things. 🚀

The conversation continues at AI Skills Fest. From interactive learning to real-world AI use cases, today's sessions are packed with ideas and inspiration for learners, professionals, and organizations worldwide. Which one are starting first? ⬇️ Turn your everyday work into credentials that count 6/9 5:00 PM PDT (Americas) 6/9 7:30 IST (EMEA/Asia) Prepare for Microsoft Certification Exam AB-730: AI Business Professional 6/9 6:30 PM PDT (Americas) 6/9 9:00 IST (EMEA/Asia) Microsoft Cert Exam Prep for DP-700: Data Engineering with Microsoft Fabric 6/9 8:30 AM IST (EMEA/Asia) 6/9 10:00 AM PDT (Americas) And when you finish, you can jump into a playlist specifically designed for your role: