Cole Medin(@cole_medin):SWE-bench Verified is contaminated. OpenAI just published the proof. Top models: 70%+ on Verified, ~23% on SWE-bench Pro. All frontier models can reproduce original fixes from memory. 59% of hard tasks have flawed tests. The benchmark everyone was citing? Meaningless now.

Cole Medin

@cole_medin

All about entrepreneurship, AI, and coding 💻 Combining my passions to build an awesome business and teach others my secrets 👨‍🏫

가입 October 2022

89 팔로잉 중 2.6K 팬

Cole Medin@cole_medin

2026.02.25 22:39

SWE-bench Verified is contaminated. OpenAI just published the proof. Top models: 70%+ on Verified, ~23% on SWE-bench Pro. All frontier models can reproduce original fixes from memory. 59% of hard tasks have flawed tests. The benchmark everyone was citing? Meaningless now.