# Learning Palantir Foundry
🚀 Are you recomputing billion-row tables in full every single day? Process only the delta, and your compute costs drop dramatically.
📌 Title and Feature URL
Title: Incremental Transforms
URL:
📝 Overview
Incremental transforms enable efficient data processing by handling only the data added or changed since the last run, instead of reprocessing the entire dataset. They're enabled with the `
@incremental()` decorator, which automatically chooses between incremental and snapshot execution based on how the inputs changed.
🔧 How It Works
The `
@incremental()` decorator wraps a transform function to give it delta-processing capability.
- It converts the standard input/output objects into incremental variants: `IncrementalTransformInput`, `IncrementalTransformOutput`, and `IncrementalTransformContext`
- Input read modes can be `added` (new rows since last run, the default), `previous` (state from the last run), or `current` (the full current dataset)
- Output write modes are `modify` (append to existing output) or `replace` (overwrite entirely); the default is `modify` for incremental runs and `replace` for snapshot runs
- Key parameters include `require_incremental` (fail if incremental isn't possible), `semantic_version` (bumping it triggers a snapshot rebuild), `snapshot_inputs` (exempt specific inputs from incremental constraints), and `strict_append` (enforce append-only safety)
🛠 Practical Usage
- Add `
@incremental()` to large append-heavy log or transaction tables to replace daily full recomputes with delta processing
- When you change logic, bump `semantic_version` to safely trigger a snapshot rebuild
- Use `require_incremental` to force delta execution when you don't want a silent full reprocess
- Use `strict_append` when you need strict append-only guarantees
🎯 Use Cases
- Slashing soaring compute costs from daily full recomputes of billion-row tables via delta processing
- Serving as the core cost-optimization technique that determines the economics of large-scale projects
- Daily ingestion of append-only transaction histories and event logs
- Streamlining pipelines whose upstream grows only through additions (APPEND/UPDATE)
⚠️ Caveats
- Preview features always run non-incrementally
- Unless requirements are met (all non-snapshot inputs contain additions only via APPEND/UPDATE, the input list stays stable, `semantic_version` is unchanged, etc.), the transform automatically runs in snapshot mode and fully replaces the output
- Updated or deleted input files must be marked as snapshot inputs
- The `previous` mode requires schema validation matching the previous output structure
- Transform logic must support both incremental and snapshot execution paths
#
PalantirFoundry# #
DataEngineering#