Data skew is not always about row counts.
Snowpark lets engineers run Python, Java or Scala UDFs natively on Snowflake's query engine. At scale, two workers can process the same number of rows while one does far more compute.
DySkew detects that at runtime and redistributes dynamically. No tuning required on your end. Result: ~30% lower P99 UDxF execution time in production, P90 down 27% on large warehouses.
How we built it:
Data quality is only part of the equation.
When AI is being trained and evaluated, there are a few questions worth asking:
- Can each contribution be traced?
- Who contributed the data?
- Was their domain expertise verified?
- Is there a verifiable audit trail of how it was produced and validated?
For high-stakes AI, “high-quality” data isn’t enough.
It needs to be expert-validated, traceable, and auditable.
Data engineering agents need harder evals than “write this SQL query.”
data-eng-bench gives agents a live dbt project, an enterprise-scale warehouse, build/fix tasks and hidden verifier tests that check whether the resulting pipeline actually works.
Now open source: