DuckLake: The Lakehouse That's Just SQL & Parquet [Talk Python to Me, Ep. 562] How many files does your query read before it reads any data? On some data lakes, you go through JSON and metadata files first, just to learn which Parquet files matter. #DuckLake# asks one SQL question instead. The metadata lives in a real database. The data stays in plain Parquet. That's the entire format. With Quack as the catalog, DuckLake handles 200 transactions a second under heavy contention. No other open table format comes close. In this episode of @TalkPython, host @mkennedy talks with @holanda_pe, lead developer for DuckLake at DuckLabs, and Guillermo Sanchez Dionis, who works on both DuckLake and the new Quack protocol – together they discuss all the details of DuckLake, Quack, and more. Watch the episode here or listen wherever you get your podcasts:

Ducklake with duckdb is really really good. Tell your agent to use it if you ask it to import a lot of data ( like health data from Fitbit ). If you use Hermes, just go “/learn duckdb and ducklake and use it” and all your data asks will get better.

A week from today, join Alex Monahan and Matt Martin to cover all things DuckLake performance optimization. Their new chapter is live, so they'll cover how to: ​🦆Find the bottleneck first with metadata logging, EXPLAIN ANALYZE, and JSON query profiles (and why handing those profiles to an LLM works well) ​🦆Pick partition keys and sort orders so queries skip whole files and row groups, and spot when bucket partitioning works against you ​🦆Sort on two columns at once with space-filling curves, something the Iceberg spec doesn't allow ​🦆Tune compression, file size, and row group size together instead of one at a time ​🦆Keep the catalog and storage lean with inlining, compaction, snapshot expiration, and cleanup, and know which of these actually make queries faster Register here to join them:

DuckDB Monthly #46# is out. Spotlight: @marklit82, of 1.1 Billion Taxi Rides fame, who has been showing what DuckDB can do for GIS since 2023. Plus Jev typed AI answers in SQL, DuckDB 2.0's PEG parser and async S3 reads, and DuckLake maintenance.

Just listed: one open-format lakehouse, move-in ready for your agents. Open formats, Parquet on object storage, ACID transactions, snapshots you can revisit, and a schema that evolves with you. Storage and compute are billed separately. Utilities not included. The open house is Nov. 4 at Data Outpost, where @__AlexMonahan__ (MotherDuck) and Matt Martin (State Farm) are running "Lakehouses for Robots: Scaling DuckLake and Iceberg for Agent Workloads." You'll query Iceberg and DuckLake from @DuckDB and MotherDuck with the agent of your choice, and learn to get the most out of your local laptop so you can build faster. Performance tuning and agent compartmentalization keep your agents' answers quick and safe. Whether you're new to lakehouses or a seasoned expert, you'll leave with a working lakehouse example. Bring a laptop. 🎟️ Tickets:

What’s Next for DuckDB? 🦆🤔 Hannes Mühleisen, co-creator of DuckDB, was recently interviewed by Kris Jenkins of the Developer Voices podcast on every challenge, reward, twist, and turn of building a database as popular as DuckDB. From the early days of the product to the most recent acquisition of @ducklabs_com by @awscloud, Hannes – and what it means for the future of the open source DuckDB, DuckLake, and Quack projects*. It’s an interview as much as a story – an odyssey of what it means to build not only an open source product, but also a community, a company, and what success looks like in the day to day journey along the way. 📺 Watch the video episode (or read the transcript) here: 🎧 Or, listen wherever you get your podcasts. *Spoilers: DuckDB, DuckLake, Quack, and other open source projects in the DuckStack will remain open source and the project’s intellectual property, MIT license, and trademark are controlled by the DuckDB Foundation. The DuckLabs company is joining AWS, but the OSS projects remain OSS and controlled solely by the Foundation. 🦆 🦆