Pydantic(@pydantic):Airbnb argues the "Biggest Brains" read more than 100 traces before writing a single evaluator. Their rule: an eval written before you've seen the failures mostly measures the author's imagination. Reading traces means all of them. The database call, the upstream timeout, the browser session that actually broke. Braintrust sees the slice you mail in. Your observability stack sees what happened. The eval score points at the model. The trace is the crime scene. We implemented Airbnb's whole framework where your system already lives: three layers, judge calibration against human gold sets, production sampling. All of it working code: https://t.co/fV1hadSYFU Day four: @strawgate remains in charge. Tomorrow: the benchmark.

2026.08.06 21:03

Airbnb argues the "Biggest Brains" read more than 100 traces before writing a single evaluator. Their rule: an eval written before you've seen the failures mostly measures the author's imagination. Reading traces means all of them. The database call, the upstream timeout, the browser session that actually broke. Braintrust sees the slice you mail in. Your observability stack sees what happened. The eval score points at the model. The trace is the crime scene. We implemented Airbnb's whole framework where your system already lives: three layers, judge calibration against human gold sets, production sampling. All of it working code: Day four: @strawgate remains in charge. Tomorrow: the benchmark.