The best coding-agent harnesses solve half the tasks per attempt, but only a third every time.
@jeffweisbein asked on my launch post how often each harness solves the same task on all three attempts. That matters when an agent runs without anyone watching.
The Harness Leaderboard now shows it. It compares nine coding-agent harnesses on the same open models.
Averaged over three open models, mini-swe-agent (@KLieret) solves 51.4% of tasks per attempt, 68.6% at least once in three tries, and 33.0% every time.
Five harnesses solve 32.0% to 33.3% every time, too close to call. @openclaw solves 15.3%.
On Flint, the weakest model, Hermes Agent (@NousResearch) solves 28.6% at least once but only 10.0% every time.
Leaderboard:
顯示更多
