Henry Zhang(@henryzhangumich):Just completed a full end to end run of the full 113 task DeepSWE benchmark on ox-alpha. The Rumored ~80% pass rate is completely incorrect. Actual benchmark result is 58.4%, landing the model at almost identical performance to Claude Opus 4.8 (59%) 🧵

Henry Zhang

@henryzhangumich

inference @cohere | prev @amazon | cs + biochem @umich | views are my own

参加 February 2025

494 フォロー中 685 ファン

Henry Zhang@henryzhangumich

2026.08.22 07:33

Just completed a full end to end run of the full 113 task DeepSWE benchmark on ox-alpha. The Rumored ~80% pass rate is completely incorrect. Actual benchmark result is 58.4%, landing the model at almost identical performance to Claude Opus 4.8 (59%) 🧵