Yafah Edelman(@YafahEdelman):We audited 15 benchmarks and labeled 9 flawed: - In Terminal Bench 4.0 we found 45.5% of tasks to be broken after reviewing github issues. - In HLE, 46% of the 48 questions we randomly sampled were broken. - In DeepSWE 1.1 we found a bug that can break grading for every task.

Yafah Edelman

@YafahEdelman

Chief Strategy Officer @EpochAIResearch she/her

Joined March 2014

618 Following 2.4K Followers

Yafah Edelman@YafahEdelman

2026.09.17 22:49

We audited 15 benchmarks and labeled 9 flawed: - In Terminal Bench 4.0 we found 45.5% of tasks to be broken after reviewing github issues. - In HLE, 46% of the 48 questions we randomly sampled were broken. - In DeepSWE 1.1 we found a bug that can break grading for every task.

640

Forward to community

Most Popular Users