Evals are great but in the real world it’s hard for me to really know which frontier model to use I switched 100% to Astra recently but it burned me out by overwriting core files multiple times, and by always claiming “it wasn’t me” Back to Opus 5.5 today Honestly hard to know

Evals & Harnesses beyond Code Mode & Pass Rate: in a Math degree, assignments & exams are largely closed book + no fancy calculator this is for good reason tooling can help you get answers, but putting constraints on your tools helps you sharpen innate skills developing reasoning skills on the underlying structure of problem solving is often more valuable to learn than how to use a tool this imo totally applies to LLM RL training & the current code mode paradigm I LOVE code mode and find it amazing how well models can use code to solve any problem, but using code as a tool is a decision on giving a model a very powerful harness to solve problems, Python is Turing Complete ofc this has tradeoffs such as not being able to solve other problems Im seeing this pretty distinctly in visual reasoning tasks that don’t have code mode, models just aren’t great at persistence or visual perception without code Maybe that’s a fine tradeoff, but AGI level should be able to do tasks without code and I bet training in harnesses without code mode will make them generally more intelligent

Local Evals v0.1.0 released 🥳 Worked on refining the Eval run setups today to make it easier to use for newbies like me Here's the repo if you want to play around:

AI evals have been the "it" skill for product teams for over a year. I've even called evals a new discovery habit. But I still meet product teams who only have a vague idea of what evals are. And it's not their fault. Most of the writing on this topic is intended for engineers or just isn't specific enough. I recently created an in-depth eval guide to explain what evals are and why product teams can and should create them. I did my best to make it practical, hands-on, and easy to follow. AI evals (short for evaluations) are methods for measuring whether an AI product or workflow is performing well. Evals give teams confidence that their AI applications are doing what they expect them to do. They help teams maintain quality and catch issues before they reach users. Similar to other discovery habits like interviewing and assumption testing, evals can act as a feedback loop to ensure we are on the right track. If you want to learn more about this new discovery habit, explore my new guide:

Building evals is hard, and only getting harder, but I also think it's some of the most interesting and exciting stuff I've ever done. Going much deeper with my eval suites at @VulcanBench, very excited to share what's next. But first, within the next hour, I will be sharing the results from my first Harness Benchmark, really excited about this one, and it was hard in a different way since I had to keep updating guardrails. Now, I have what I think is one of the cleanest ways to benchmark not just a model itself, but a model in a harness. Stay-tuned!

@hud_evals @ycombinator Tera — zero-token browser use solve once, reuse forever: a browser RL env that learns from past traces, then replays matched workflows with 0 LLM tokens. took #3# overall.