“I used my agent to formally verify my software and it fixed all these bugs” is the new “I asked an LLM judge to fix my LLM output”. The devil is in the details and it is insanely difficult to get the formalisms “right” (eg interpretable, expressive, extensible) for humans and agents to co-create software. Seems that we are in for an entirely new age of learning how to build software

Very exciting to see how much this post resonated with the community. The uncomfortable dialectic is that (1) evals are necessary to build good agents, but also (2) agents are extremely helpful (and necessary) for automating tedious parts of the evals lifecycle. Human attention and labeling simply cannot scale up to the high amount of unstructured info that is agents. Unfortunately nobody has cracked the most perfect and efficient workflow to do evals. We share our thoughts on our current iteration in this post!!

Releasing our runtime dynamic compression framework that achieves 1.5-2.0 bit compression at high quality. This is integrated into the bitsandbytes2 library, which starts as a private beta today. Paper: Private beta signup:

Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp took its automatic receipt collection from 35% to 83% accuracy. — @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced. — @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score. — @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost. So I asked the 🐐s of evals, @HamelHusain and @sh_reya, to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting. Read it here:

jev is a new type of model - "decision models" there are a few open source alternatives, one is SemIf (based on qwen3.5) we're hosting SemIf and offering it for free through LangSmith Gateway for the next week!

Starting tomorrow, our lab will hold an open-source week: 2 software frameworks, 4 papers, all building a coherent ecosystem. The theme: Frontier AI on Hardware You Own Blog post: SoTA results in: Autocompaction Autonomous Research Model compression Test-time scaling Deep Research And a healthcare RL environment giving you a new level of complexity to train healthcare agents. The ecosystem that we will release is built to be as usable as possible. Agent sessions that run overnight and go on for tens of millions of tokens is made easy. Model compression of a model is automatic: you just get a good model that runs fast locally -- no expertise required. An autonomous research that works out of the box. Our ecosystem enables a new level of work that can be done locally.

We re-worked my Jev/DuckDB example, and boy, was my original vibecoded extension excruciatingly inefficient. At LEAST 20x-40x faster now, often way more. Jev is going to revolutionize analytics. Something very cool to share tomorrow :)

Can you use Jev for Evals? Yes! Remember that a LLM Judge is also classifier*. Make sure to test your classifiers against human labels and don't overfit. Hope this helps! *

Q: How do you review a trace that is really large? A: Focus on the first upstream failure. Make the relevant evidence easy to inspect, with details reviewers can expand as needed.

got access to Jev!! heard a lot of good things about it, especially as a classifier. really curious to try it for evals. lots of llm-as-a-judge for binary decisions are going to get replaces. give Jev some state + a bunch of questions and get back probabilities you can actually threshold on. also feels pretty useful inside agent harnesses as a control plane for making all those fuzzy decisions like which model or tool to route to, whether to continue or retry, whether an output looks good enough, whether something needs human review, etc. basically letting the llm handle generation, while Jev handles a lot of the judgment calls around it.

I made a DuckDB extension where you can use @typesafeai 's Jev to do quick classification of rows in any csv/parquet file or duckdb table about 10sec for 1k rows ~ better than using an LLM, way more ergonomic than a classifier game-changing for data analysis!

Unpopular opinion—we should not publish system design papers any more (given that most papers have "untold story", instead, we should focus on (1) the problem (2) why the solution from AI does not work. How many disagree?

TMLR has faced a deluge of submissions, necessitating stricter desk rejection policies due to limited reviewer capacity Co-EiC Nihar Shah reached out to authors of 10 papers slated for desk reject. Could they answer questions about their *own* submission?

Just updated our AI evals FAQ with 5 new questions, 48 questions & answers total! New FAQs just added: - Do I need a reference answer or rubric before annotating data? - How can I do evals when traces contain sensitive data? - How do you review a trace that is really large? - How much context should I give a LLM judge? - What should I do when my “gold” eval dataset becomes stale? It's all here: