# Learning Palantir Foundry 🚀 Answer "where did this dashboard number come from?" in an instant. Data Lineage is an exploration tool that visualizes the entire flow of your data. 📌 Title and Feature URL Title: Data Lineage URL: 📝 Overview Data Lineage is an interactive visualization tool that comprehensively shows how data flows through the Foundry platform. It helps you understand data movement, dependencies, and transformations across your entire data ecosystem. Because you can trace the lineage from sources through pipelines, the Ontology, and apps as a graph, it sharply reduces the cost of incident response and audit explanations. 🔧 How It Works It represents data dependencies through a graph-based visualization. - Find datasets using project names, table identifiers, or row labels, and browse data directly from Foundry Projects - Expand or collapse ancestor (upstream) and descendant (downstream) relationships for any dataset - View multiple table attributes at once, down to schema details, build timestamps, and source code - Apply custom color schemes to highlight pipeline characteristics such as stale datasets - Create shareable pipeline snapshots to communicate within the team 🛠 Practical Usage - Trace upstream from a dashboard or output dataset to pinpoint the source of a number - When an upstream schema changes, trace downstream to map the blast radius - Color-code stale datasets to discover neglected pipelines - Drill down from a high-level overview into granular technical details like transformation code and execution history - Share pipeline snapshots to document data workflows across functions 🎯 Use Cases - Instantly answering "what is the origin of this dashboard's number" - Identifying the impact scope of upstream schema changes in advance to prevent incidents - Presenting data lineage during audits to cut explanation costs - Finding stale or unused datasets to tidy up pipelines ⚠️ Caveats - This overview page does not explicitly discuss performance or graph-complexity constraints with extremely large pipelines - Lineage covers data flow within the Foundry platform; processing outside the platform is out of visualization scope - The accuracy of lineage depends on transforms and pipelines being properly configured within Foundry #PalantirFoundry# #DataLineage#

# Learning Palantir Foundry 🚀 Are you recomputing billion-row tables in full every single day? Process only the delta, and your compute costs drop dramatically. 📌 Title and Feature URL Title: Incremental Transforms URL: 📝 Overview Incremental transforms enable efficient data processing by handling only the data added or changed since the last run, instead of reprocessing the entire dataset. They're enabled with the `@incremental()` decorator, which automatically chooses between incremental and snapshot execution based on how the inputs changed. 🔧 How It Works The `@incremental()` decorator wraps a transform function to give it delta-processing capability. - It converts the standard input/output objects into incremental variants: `IncrementalTransformInput`, `IncrementalTransformOutput`, and `IncrementalTransformContext` - Input read modes can be `added` (new rows since last run, the default), `previous` (state from the last run), or `current` (the full current dataset) - Output write modes are `modify` (append to existing output) or `replace` (overwrite entirely); the default is `modify` for incremental runs and `replace` for snapshot runs - Key parameters include `require_incremental` (fail if incremental isn't possible), `semantic_version` (bumping it triggers a snapshot rebuild), `snapshot_inputs` (exempt specific inputs from incremental constraints), and `strict_append` (enforce append-only safety) 🛠 Practical Usage - Add `@incremental()` to large append-heavy log or transaction tables to replace daily full recomputes with delta processing - When you change logic, bump `semantic_version` to safely trigger a snapshot rebuild - Use `require_incremental` to force delta execution when you don't want a silent full reprocess - Use `strict_append` when you need strict append-only guarantees 🎯 Use Cases - Slashing soaring compute costs from daily full recomputes of billion-row tables via delta processing - Serving as the core cost-optimization technique that determines the economics of large-scale projects - Daily ingestion of append-only transaction histories and event logs - Streamlining pipelines whose upstream grows only through additions (APPEND/UPDATE) ⚠️ Caveats - Preview features always run non-incrementally - Unless requirements are met (all non-snapshot inputs contain additions only via APPEND/UPDATE, the input list stays stable, `semantic_version` is unchanged, etc.), the transform automatically runs in snapshot mode and fully replaces the output - Updated or deleted input files must be marked as snapshot inputs - The `previous` mode requires schema validation matching the previous output structure - Transform logic must support both incremental and snapshot execution paths #PalantirFoundry# #DataEngineering#

# Learning Palantir Foundry 🚀 Bring complex logic that no-code can't reach into your data platform, along with full software-engineering quality control. That's what Code Repositories delivers. 📌 Title and Feature URL Title: Code Repositories (Python Transforms) URL: 📝 Overview Code Repositories is a web-based integrated development environment (IDE) for creating and collaborating on production-ready code within Foundry. It provides a friendly UI over the underlying Git repositories, so teams can work without command-line access. With platform-specific features, you can apply software development practices directly to data engineering. 🔧 How It Works Version control and collaboration are at its core. - Common Git tasks (branching, committing, release tagging) execute through the web UI - Pull requests drive code review, with "highly configurable" permissions that support quality assurance such as mandatory reviews - IntelliSense, linting, error checking, and contextual help dialogs are available across all repository types - Transforms repositories let you author data transformation logic in Python, Java, or SQL with preview and debugging - Functions repositories natively integrate the Ontology and run low-latency business logic in TypeScript or Python 🛠 Practical Usage - Use PySpark to implement billion-row entity resolution and complex business rules in code - Require PR reviews so a second reviewer and CI checks must pass before merge - Add unit tests to guard transform logic against regressions - In Functions repositories, leverage Ontology-data-type autocomplete to write logic safely - Bring machine learning workflows into the platform via model development repositories 🎯 Use Cases - Implementing complex reconciliation and business rules in PySpark that Pipeline Builder can't express - Structurally eliminating "regressions from editing production directly" through mandatory reviews and branch-based workflows - Implementing derived KPIs and validation logic as Functions reused across apps - Managing ML model training and inference code under governance ⚠️ Caveats - The docs note that Japanese translations are machine-generated and unverified, so localized content may have accuracy limitations - Each repository type (Transforms/Functions/Model) supports different languages and purposes, so pick the one that fits your goal - Being a pro-code environment, the quality benefits only materialize if your organization establishes review, CI, and test practices #PalantirFoundry# #DataEngineering#

# Learning Palantir Foundry 🚀 "How do we connect to an on-prem Oracle or SAP in a closed network without poking holes in the firewall?" Data Connection is what clears that first hurdle of enterprise adoption. 📌 Title and Feature URL Title: Data Connection URL: 📝 Overview Data Connection is an application that synchronizes external system data into Foundry for use across the data integration, modeling, and ontology layers. It also supports outbound connections through webhooks and data exports to write data back to external systems. It handles many source types and abstracts away the messy parts — authentication, scheduling, and monitoring — so you can configure pipelines from simple interfaces. 🔧 How It Works Foundry standardizes data connections around three principles: - Robustness: automatic retries, processing in small batches, and integrated health monitoring that warns of failures. Data should be ingested "as-is" from the most primitive source, making Foundry's versioned pipelines the single source of truth for all transformations rather than depending on external preprocessing. - Extensibility: beyond standard integrations (databases, FTPS, HDFS, S3, SFTP), the system accommodates new source types. Because core functions like scheduling and orchestration are standardized, only connection-specific adjustments are needed. - Usability: the system abstracts complexity, letting users configure through simple interfaces instead of manually managing authentication, scheduling, and monitoring. - Key components include agent setup, source configuration, batch/streaming syncs, webhooks, and exports. 🛠 Practical Usage - Access Data Connection from the workspace navigation or the application portal. - Configure a source, then set up a batch or streaming sync to ingest data "as-is." - Concentrate post-ingestion transformations in Foundry pipelines, avoiding preprocessing on the source side. - For write-back, configure outbound integrations using webhooks or exports. 🎯 Use Cases - Connect on-prem Oracle/SAP in a closed network via an agent model (outbound-only) without firewall changes. - Scheduled batch ingestion from diverse sources such as databases, SFTP, and S3. - Outbound integration to write processed results back into external systems. ⚠️ Caveats - The design assumes ingesting data "as-is" and centralizing transforms in Foundry pipelines; transforming on the source side undermines traceability. - Configuring agents and sources requires proper network and authentication setup. - Do not assume external service limits or terms; verify the constraints of each connected source in advance. #PalantirFoundry# #DataIntegration#

# Learning Palantir Foundry 🚀 "Sales reps see only the rows for their assigned customers" — achieved on a single dataset, without spawning a copy per department. That is what Restricted Views (row-level security) deliver. 📌 Title and Feature URL Title: 制限付きビュー(行レベルセキュリティ) URL: 📝 Overview Restricted Views implement granular row-level access control. A restricted view is built on top of a backing dataset and lets different users see different subsets of the same underlying data based on defined permissions. This removes the need to copy datasets per team: you keep one dataset as the source of truth shared across the company while splitting visibility at the row level. 🔧 How It Works Restricted views operate through policies containing rules that determine row visibility. - Policies evaluate the viewing user's attributes, column names from the backing dataset, and specific values (strings, Booleans, numbers, arrays) to decide which rows are shown. - When referencing users, groups, or organizations, you must use the unique identifier (UUID) in both the policy column and the policy definition — names alone will not work. - In marking-backed views, the upstream dataset holds a STRING ARRAY column of Marking IDs, and each row is visible only to users with the required markings. - A restricted view is built on top of a backing dataset and cannot be used as an input for transforms. - Experimental branching support allows adding and merging restricted view policy changes. 🛠 Practical Usage - Add a column to the backing dataset that drives row access (for example, assigned branch or organization ID). - Define a policy that matches that column against user attributes to build the row-level filter. - Save restricted views in a separate Project from the source datasets to keep access management clean. - For markings, attach an array of required Marking IDs per row to control visibility. 🎯 Use Cases - Limit sales representatives to viewing customers at their assigned branch. - Separate records on a shared table by department or organization. - Disclose differently classified records only to users holding the required markings. ⚠️ Caveats - Restricted views cannot serve as transform inputs, so they cannot be plugged directly into downstream pipeline processing. - Users, groups, and organizations must be referenced by UUID; name-based references do not work. - Merging policy changes via branching is experimental and may not be universally available. #PalantirFoundry# #DataGovernance#