Ofir Press(@OfirPress):Cool multi-agent experiments in the 5.5 system card! They're getting a solve rate of close to 100% on their subset of 166 ProgramBench instances (out of 200), so I think they're not using the tougher programs (FFmpeg, the PHP compiler, ...) since we're seeing a very low solve rate on those.

Ofir Press

@OfirPress

SWE-bench, ProgramBench, SWE-sweep. Postdoc @Princeton. PhD @nlpnoah @UW.

参加 June 2016

10K フォロー中 20.4K ファン

Ofir Press@OfirPress

2026.09.22 19:58

Cool multi-agent experiments in the 5.5 system card! They're getting a solve rate of close to 100% on their subset of 166 ProgramBench instances (out of 200), so I think they're not using the tougher programs (FFmpeg, the PHP compiler, ...) since we're seeing a very low solve rate on those.

John Yang@jyangballin

2026.09.22 19:33

Multi-agent runs on ProgramBench in Opus 5.5 system card Multi-agent systems get to the same outcomes as a single-agent, but quicker. Worth noting that just 166/200 tasks were used. I still strongly encourage using all 200! The long tail of hard tasks is really difficult.