註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Ofir Press
@OfirPress
SWE-bench, ProgramBench, SWE-sweep. Postdoc @Princeton. PhD @nlpnoah @UW.
加入 June 2016
10.3K 正在關注    20.8K 粉絲
GPT-5.6 Sol and Luna were built by roughly the same people using roughly the same infra, I don't think we can blame the 20 point difference in Luna just on memorization. There's something else being tracked here in addition to memorization. Maybe MaxxBench questions are slightly more 'out of domain' than DeepSWE questions?
顯示更多
Nice DeepSWE score! Does it survive a lowercase filename? Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement. Models clustered around 74% on DeepSWE end up 23 pp apart. Good at coding, or good at DeepSWE? 🧵
顯示更多
0
14
59
0
轉發到社區