OpenAI(@OpenAI):We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability. We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as a leading coding eval.

OpenAI

@OpenAI

OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity. We’re hiring:

Joined December 2015

4 Following 5.4M Followers

OpenAI@OpenAI

2026.07.08 21:41

We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability. We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as a leading coding eval.