Great initiative to eval the evals. It incentivizes the industry to build genuinely high-quality benchmarks.
Ty for auditing PTB and digging up the old SimpleQA too :)
Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.