I was excited to hear about this initiative, but a bit disappointed in the rigor -the “task flaws” would affect < 3% of rollouts on the leaderboard -the impact on the final agent scores would not exceed our reported confidence intervals we publish the raw receipts for all our leaderboard runs (per trial logs / trajectories / rewards) so this is pretty easy to figure out. It is also pretty easy to find new task issues (that is why we release the receipts!). we love when the community reports (new) issues. no new issues were opened as part of the report - the “flaws” are simply pulled from issues triaged by the TB maintainers from our public GitHub repo. fwiw 100% of terminal-bench tasks are flawed (all software has bugs). what matters is the magnitude of the impact of those bugs and the mechanism that the benchmark creators have for continuous detecting and addressing bugs. this is the motivation behind our “continuous benchmarks” methodology. I’m glad that epoch is starting an important conversations on benchmark quality. However, auditing a benchmark should not be a one-time thing: coming together as a community to build public tools and define practices for task CI / CD is more in service of the goal of enforcing good benchmark practices. we have been experimenting and pushing the possibilities of these benchmark CI / CD tools and practices as part of the terminal-bench project and are open wide collaborations with all who are interested by that problem