Cole Medin(@cole_medin):SWE-bench Verified is contaminated. OpenAI just published the proof. Top models: 70%+ on Verified, ~23% on SWE-bench Pro. All frontier models can reproduce original fixes from memory. 59% of hard tasks have flawed tests. The benchmark everyone was citing? Meaningless now.

Cole Medin

@cole_medin

All about entrepreneurship, AI, and coding 💻 Combining my passions to build an awesome business and teach others my secrets 👨‍🏫

加入 October 2022

89 正在关注 2.6K 粉丝

Cole Medin@cole_medin

2026.02.25 22:39

SWE-bench Verified is contaminated. OpenAI just published the proof. Top models: 70%+ on Verified, ~23% on SWE-bench Pro. All frontier models can reproduce original fixes from memory. 59% of hard tasks have flawed tests. The benchmark everyone was citing? Meaningless now.

显示更多