# Learning Palantir Foundry 🚀 Are you recomputing billion-row tables in full every single day? Process only the delta, and your compute costs drop dramatically. 📌 Title and Feature URL Title: Incremental Transforms URL: 📝 Overview Incremental transforms enable efficient data processing by handling only the data added or changed since the last run, instead of reprocessing the entire dataset. They're enabled with the `@incremental()` decorator, which automatically chooses between incremental and snapshot execution based on how the inputs changed. 🔧 How It Works The `@incremental()` decorator wraps a transform function to give it delta-processing capability. - It converts the standard input/output objects into incremental variants: `IncrementalTransformInput`, `IncrementalTransformOutput`, and `IncrementalTransformContext` - Input read modes can be `added` (new rows since last run, the default), `previous` (state from the last run), or `current` (the full current dataset) - Output write modes are `modify` (append to existing output) or `replace` (overwrite entirely); the default is `modify` for incremental runs and `replace` for snapshot runs - Key parameters include `require_incremental` (fail if incremental isn't possible), `semantic_version` (bumping it triggers a snapshot rebuild), `snapshot_inputs` (exempt specific inputs from incremental constraints), and `strict_append` (enforce append-only safety) 🛠 Practical Usage - Add `@incremental()` to large append-heavy log or transaction tables to replace daily full recomputes with delta processing - When you change logic, bump `semantic_version` to safely trigger a snapshot rebuild - Use `require_incremental` to force delta execution when you don't want a silent full reprocess - Use `strict_append` when you need strict append-only guarantees 🎯 Use Cases - Slashing soaring compute costs from daily full recomputes of billion-row tables via delta processing - Serving as the core cost-optimization technique that determines the economics of large-scale projects - Daily ingestion of append-only transaction histories and event logs - Streamlining pipelines whose upstream grows only through additions (APPEND/UPDATE) ⚠️ Caveats - Preview features always run non-incrementally - Unless requirements are met (all non-snapshot inputs contain additions only via APPEND/UPDATE, the input list stays stable, `semantic_version` is unchanged, etc.), the transform automatically runs in snapshot mode and fully replaces the output - Updated or deleted input files must be marked as snapshot inputs - The `previous` mode requires schema validation matching the previous output structure - Transform logic must support both incremental and snapshot execution paths #PalantirFoundry# #DataEngineering#