注册并分享邀请链接,可获得视频播放与邀请奖励。

Ofir Press
@OfirPress
SWE-bench, ProgramBench, SWE-sweep. Postdoc @Princeton. PhD @nlpnoah @UW.
加入 June 2016
10.3K 正在关注    20.8K 粉丝
GPT-5.6 Sol and Luna were built by roughly the same people using roughly the same infra, I don't think we can blame the 20 point difference in Luna just on memorization. There's something else being tracked here in addition to memorization. Maybe MaxxBench questions are slightly more 'out of domain' than DeepSWE questions?
显示更多
Nice DeepSWE score! Does it survive a lowercase filename? Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement. Models clustered around 74% on DeepSWE end up 23 pp apart. Good at coding, or good at DeepSWE? 🧵
显示更多
0
14
59
0
转发到社区