Quick database rant :) There is still a lot of basic wood to chop when it comes to agent observability. It's a totally new database workload. OLAP systems are built for <5kb events with UUIDs and very repetitive dictionary-like strings. And the techniques that work to query those do not work for agent traces. That's compounded by the crazy cost and scale problems. Agents have PMF now and so it's common for these workloads to generate GB/s per agent of tracing data. It's just not practical operationally to store this on attached disks. You have to use object storage. This problem of fat rows + object storage throws the existing systems' query schedulers for a total loop. They're used to reading local data in microseconds and for that data to be very dense / high signal. If you're searching for random shit in agent traces, this falls apart completely, and you end up doing a terrible job of scheduling work on your CPUs. Which in turn means you need to create a lot of OS threads to handle the workload and make your kernel very unhappy. We waddled our way around this by cobbling together Tantivy and our own I/O scheduler for a while, but ultimately realized we had to rebuild the execution engine from scratch. We tried using some off the shelf stuff (Datafusion is by far the most promising system here), but sadly to achieve optimal performance you need control flow while decoding postings from the inverted index so that you can short circuit. Tantivy, etc. do this but the logic is interleaved with the I/O itself, so you end up with terrible cold starts no matter what. A simple way to think about this is a graph traversal on the inverted index. The columnstore-maxxing approach is to read the whole graph and then crunch on it very fast (and you can schedule that efficiently, of course). The classic inverted index approach is traverse and potentially short circuit as you do so, interleaving those operations. We solved this by building a new engine called Nitro, which does async I/O scheduling, but with control flow / short circuiting built into the actual I/O layer, so that we can still saturate the network but avoid reading stupid amounts of data before actually executing a search. For example, if you search for something like "aha! you must be right about describe_all_queries() not working for Monday's cohort", you do not want to read all postings for "you", "must", "be", "right", "not", and "for". I think the benchmarks speak for themselves.. We compared Nitro against two modern columnar systems used in agent observability. We also still have a lot of work to do here. If this kind of stuff interests you, we are hiring!

Agree and we are doing this ourselves and helping our users do it too. An easy way to setup and use agent tracing data with your eval. A little cookbook on this topic and there is a lot more to come around our own learnings and more cc @rbro112

We tested where Jev holds up as a judge. Jev was fast, cheap, and highly competitive for judging groundedness. But it lagged behind models with reasoning for math and code domains. Use it for certain eval tasks, but don't throw out your LLM-as-a-judge just yet. Read more →

Jev from @typesafeai is now available as an evaluation model in Braintrust. If you have an existing scorer, just change the model in the dropdown to Jev and reduce your scoring cost by 400x. We're very excited about this and are doing more of our own evals in the coming days.

One of the things we optimized with Patterns is identifying rather than resolving issues. Identifying and monitoring an issue well is very, very difficult. But if you do it well, then just plug into Linear, Cursor, etc. and 🚀

Loop is now so good at detecting patterns that we're using it to optimize Brainstore. We pipe all of our query plans into Braintrust and found a bunch of low-level optimizations, eg a spot in our regex code where we were over-fetching duplicate ranges from object storage.

gpt-6-astra is now available in Loop and usable for pattern detection, eval creation, trace debugging, and more. i have personally found it to be amazing at debugging. it works just as well as it does in Codex, because Loop is literally built on top of Codex ;)

I’ve joined @braintrust as the first Product Manager! Very happy to be back in-person, loving the culture here, and already attached to the pups: Hendrix, Frank, Sunshine 😍 Also helped ship a few things today: Patterns: looks across your agent traces, finds issues and patterns you probably wouldn't catch manually, and suggests what to fix. Loop: an agent inside Braintrust you can delegate work to, like digging through logs, iterating on evals, building dashboards, and generally doing the tedious stuff you don’t want to do yourself. Oh, and we’re hiring a ton if you want to help AI teams build better agents: • Director, Product Eng • Software Engineers • Data Engineer • Designers • Account Execs • Head of Finance Come work with us!

Very excited to ship a major overhaul to Loop. It is now capable of running in the background, finding issues in your traces automatically, and sending you useful alerts. Loop is architected as a single agent which has access to a broad set of tools and skills, all exposed through our MCP. The stuff we shipped today are not new features. They are effectively emergent behaviors that solve a bunch of important use cases. This architecture has a number of advantages: * Features compose (eg "find issues on traces that cost more than $5", "create a dataset of compaction issues") * You can chat with anything, eg ask for more details on a pattern it found * You can recreate 100% of Loop's features with your own agent, using our MCP The last bullet is especially important to us. We do not want you to feel like you have to use our agent to access these features. Instead, we want to prove its value by building a great user experience and dealing with orchestration for you. Take it for a spin. We'd love your feedback!

> Now, a key question is how we check that the program we get out of the AI is indeed equivalent to the higher level description / numpy ... suffice it to say that testing for semantic equivalence is possible and we do it. This is the most interesting problem in CS right now