登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

Ofir Press
@OfirPress
SWE-bench, ProgramBench, SWE-sweep. Postdoc @Princeton. PhD @nlpnoah @UW.
参加 June 2016
10.3K フォロー中    20.8K ファン
GPT-5.6 Sol and Luna were built by roughly the same people using roughly the same infra, I don't think we can blame the 20 point difference in Luna just on memorization. There's something else being tracked here in addition to memorization. Maybe MaxxBench questions are slightly more 'out of domain' than DeepSWE questions?
もっと見る
Nice DeepSWE score! Does it survive a lowercase filename? Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement. Models clustered around 74% on DeepSWE end up 23 pp apart. Good at coding, or good at DeepSWE? 🧵
もっと見る