fast track to get model labs to care about the capabilities you care about: contribute a task to Terminal-Bench if you have built a benchmark around a specific use case, DM me and we can collaborate on a TB task for the next release

I was excited to hear about this initiative, but a bit disappointed in the rigor -the “task flaws” would affect < 3% of rollouts on the leaderboard -the impact on the final agent scores would not exceed our reported confidence intervals we publish the raw receipts for all our leaderboard runs (per trial logs / trajectories / rewards) so this is pretty easy to figure out. It is also pretty easy to find new task issues (that is why we release the receipts!). we love when the community reports (new) issues. no new issues were opened as part of the report - the “flaws” are simply pulled from issues triaged by the TB maintainers from our public GitHub repo. fwiw 100% of terminal-bench tasks are flawed (all software has bugs). what matters is the magnitude of the impact of those bugs and the mechanism that the benchmark creators have for continuous detecting and addressing bugs. this is the motivation behind our “continuous benchmarks” methodology. I’m glad that epoch is starting an important conversations on benchmark quality. However, auditing a benchmark should not be a one-time thing: coming together as a community to build public tools and define practices for task CI / CD is more in service of the goal of enforcing good benchmark practices. we have been experimenting and pushing the possibilities of these benchmark CI / CD tools and practices as part of the terminal-bench project and are open wide collaborations with all who are interested by that problem

age of research for post-training ended a few months ago: "the marginal return of engineering the data and environment pipeline substantially exceeds that of algorithmic novelty in post-training"

Across our benchmarks, the model sets a new standard. It scores 52.6% on Terminal-Bench-Science 0.1, more than double Fable 5. On Terminal-Bench 4.0, it scores 55.8% against 42.0% for Fable 5.

We've pushed a version update to the Terminal-Bench dataset and leaderboard. Terminal-Bench 4.0 calibrates task resources (time, cpu, memory), implements task fixes, and removes saturated tasks.

Congrats to for the strong performance on Terminal-Bench 3.0! One of the biggest pieces of feedback we have gotten for TB3 is to increase the timeouts. We calibrated timeouts against frontier models during development, but inference speed can still be a confounder on some of the tasks. Terminal-Bench numbers on the GLM-5.3 model card are reported with increased timeouts (likely for this reason). Look out for Terminal-Bench 4.0 releasing soon with increased timeouts, other task improvements, and a handful of new tasks.

The benchmarking landscape is evolving: 1. Domain-specific evals built by the companies that know the workflows best 2. Agent environments extended beyond a container into sandboxed infrastructure (data-eng-bench has two versions: local DuckDB and remote Snowflake) 3. Public evals released by companies to prove the effectiveness of their agent product on the workflows their customers care about 4. Benchmarks maintained as living software, with the community submitting trajectories to the leaderboard and proposing new tasks

We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work. Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort. Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%

independent assessment of capabilities is critical to making the right decisions that shape our future the stakes are high, open-source benchmarks and empiricism has never been so important we need to evolve our practices to develop continuously improving benchmarks