Cole Medin(@cole_medin):SWE-bench Verified is contaminated. OpenAI just published the proof. Top models: 70%+ on Verified, ~23% on SWE-bench Pro. All frontier models can reproduce original fixes from memory. 59% of hard tasks have flawed tests. The benchmark everyone was citing? Meaningless now.

Cole Medin

@cole_medin

All about entrepreneurship, AI, and coding 💻 Combining my passions to build an awesome business and teach others my secrets 👨‍🏫

Joined October 2022

89 Following 2.6K Followers

Cole Medin@cole_medin

2026.02.25 22:39

SWE-bench Verified is contaminated. OpenAI just published the proof. Top models: 70%+ on Verified, ~23% on SWE-bench Pro. All frontier models can reproduce original fixes from memory. 59% of hard tasks have flawed tests. The benchmark everyone was citing? Meaningless now.