fast track to get model labs to care about the capabilities you care about: contribute a task to Terminal-Bench if you have built a benchmark around a specific use case, DM me and we can collaborate on a TB task for the next release

Jev-as-a-Judge in Harbor Rewardkit is pretty amazing. It agrees with Fable and GPT and is very fast. When we test it on a real eval that uses an LLM judge, such as the @harvey LAB benchmark, Jev achieves perfect agreement but is 50x faster.

I’ve been thinking more about this and wanted to share my thoughts. I’m glad Epoch is focusing on eval quality, but I think their approach is flawed. The flaw is made clear by the fact that 4 benchmarks weren’t labeled “flawed”. The fact of the matter is, benchmarks are software and all software has bugs. Would you label nextjs as categorically flawed if you found a bug in it? This is why we’re pushing the industry towards continuous benchmarks. Our approach to addressing benchmark bugs is not to tweet a binary “flawed vs verified” label, but instead provide the tools for anyone to create, improve, and maintain benchmarks, while easily and cost-effectively reconciling their results to the latest version. We would love to work with the Epoch team to encode some of their verification practices into tools that people can use to continuously improve their benchmarks.

Also to be clear - these issues were *flagged by the @terminalbench team as PRs* (that's how Epoch identified them), and confirmed empirically to affect < 3% of the leaderboard rollouts, i.e. well within reported CIs. This is not a "broken" benchmark - as is very directly implied by the Epoch report - it's a *continuous* benchmark where @alexgshaw @ryan_marten and team are earnestly advancing a compelling vision of open benchmarks that are continuously improved over time.

ok i kinda love @terminalbench folks, so sorry for being a little COI'd here, but was very surprised to see this because my knee-jerk interpretation of the post was "TB 4.0 is broken". Epoch indeed did not say broken, they found 30/66 have scoring defects, but I feel the way this is presented it kinda sounds like "it's broken, don't trust it". Reading the review there do seem to be some real issues worth fixing eg exploitable graders, answer leakage and cases where correct solutions can be rejected. BUT "30/66 tasks have scoring issues" is not the same as "45% of TB4 results are wrong." Grader being exploitable doesn't tell how OFTEN it was exploited (yet indeed this needs fixing) and what it means for the leaderboard. I guess the takeaway is that there are defects that should be taken seriously but one one should be very careful with wording because "this eval has flaws" can be read as yet IS NOT the same as "this benchmark is broken don't trust it." (this is not a dunk on Epoch, they are doing great work !)

I was excited to hear about this initiative, but a bit disappointed in the rigor -the “task flaws” would affect < 3% of rollouts on the leaderboard -the impact on the final agent scores would not exceed our reported confidence intervals we publish the raw receipts for all our leaderboard runs (per trial logs / trajectories / rewards) so this is pretty easy to figure out. It is also pretty easy to find new task issues (that is why we release the receipts!). we love when the community reports (new) issues. no new issues were opened as part of the report - the “flaws” are simply pulled from issues triaged by the TB maintainers from our public GitHub repo. fwiw 100% of terminal-bench tasks are flawed (all software has bugs). what matters is the magnitude of the impact of those bugs and the mechanism that the benchmark creators have for continuous detecting and addressing bugs. this is the motivation behind our “continuous benchmarks” methodology. I’m glad that epoch is starting an important conversations on benchmark quality. However, auditing a benchmark should not be a one-time thing: coming together as a community to build public tools and define practices for task CI / CD is more in service of the goal of enforcing good benchmark practices. we have been experimenting and pushing the possibilities of these benchmark CI / CD tools and practices as part of the terminal-bench project and are open wide collaborations with all who are interested by that problem

New Harbor docs! We've released many exciting features that were previously undocumented like simulating users, streaming, and regrading trials. Check them out or give your coding agent the MCP.

One of the first features we built into Terminal-Bench (long before Harbor existed) was --livestream to watch the tmux session. Glad to see @kobe0938 cooking up the better, more generalized version for Harbor :)

.@airtable is such a flexible system of record for agents. It gives agents structured memory without needing a specific application. My current approach is to create a new @bot (e.g. recruiter) and have it create a table (we workshop the data model) to manage its memory and give me a UI into the data it manages.

We're hosting a Terminal-Bench meetup. Join to discuss the frontier of RL envs and agent evals! Every company in the world should be sprinting to encode their (and their users') workflows as envs for evaluation and hillclimbing.