The best coding-agent harnesses solve half the tasks per attempt, but only a third every time. @jeffweisbein asked on my launch post how often each harness solves the same task on all three attempts. That matters when an agent runs without anyone watching. The Harness Leaderboard now shows it. It compares nine coding-agent harnesses on the same open models. Averaged over three open models, mini-swe-agent (@KLieret) solves 51.4% of tasks per attempt, 68.6% at least once in three tries, and 33.0% every time. Five harnesses solve 32.0% to 33.3% every time, too close to call. @openclaw solves 15.3%. On Flint, the weakest model, Hermes Agent (@NousResearch) solves 28.6% at least once but only 10.0% every time. Leaderboard:

I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference. Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent. One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53. The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint. @opencode costs $3.35 per solved task, 11.7 times as much as Pi. See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: Harness Lens:

Same weights. $50 of harness search. 3x on Terminal-Bench. We optimized the OpenCode harness for GPT-OSS-20B (high reasoning) with a meta-harness: an agent that reads failures, patches the harness, and validates the improvement. The model never changed. On held-out Terminal-Bench 2.1, the optimized harness scored 14.8%, up from OpenCode's 4.8%. That is 3.08x, from $49.97 of API credits. Three harness fixes carried most of the gain: - i7, verify before stopping: the model often edited code and ended the session without testing it. The harness gives it another turn to run the code. - i14, continue announced actions: the model would say "now run X" and then stop. The harness keeps the session going so it runs X. - i23, repair malformed tool calls: one extra `]` made OpenCode reject otherwise valid commands. The harness repairs that clear-cut error. How the meta-harness ran: 1. Start: stock OpenCode v1.18.13 + GPT-OSS-20B scored 8.5% on the dev split. 2. Iterate: Claude Code (Opus 5 max) reads the failed dev trajectories and proposes one patch. We build it and run it once per dev task, paired against the current parent. 3. Validate: candidates that clear a pre-registered margin get a 5-trial validation. They become the new parent only if their validated score surpasses the current one. 4. Stop: the loop ended after 23 iterations and about 2.4k dev attempts, when the $50 search budget was spent. Takeaway: pre-training puts capability in the weights, and post-training makes that capability usable. Harness optimization continues the same work at inference time, turning the model's behavior into more completed tasks. Same weights, 3x the finished tasks.

Codex is overoptimised for large models: it ranks 2nd out of 10 for GLM 5.2 but drops to 9th place for Gemma-4! Almost all the effort in this field goes into tuning the weights. We wanted to know how much of the final number is decided by the harness wrapped around them instead, so we ran 10 coding agent harnesses against two models on SWE-bench Pro. A lot, it turns out. Swapping the harness moves pass@1 from 23% to 52% on GLM-5.2, and from 15% to 36% on Gemma 4 26B-A4B: a wider gap than most model releases buy you. And the ranking does not transfer. The rank correlation between the two models' harness leaderboards is -0.05, which is to say: none. Codex is not alone in this. Every harness shipped by a model vendor drops on the small model — Codex 2nd to 9th, Claude Code 3rd to 7th, Qwen Code 4th to 6th — while the model-agnostic ones climb: crush 7th to 1st, opencode 8th to 2nd, pi 9th to 4th. The clearest case is crush, 7th on GLM-5.2 and 1st on Gemma 4. Run the identical scaffold on both and the small model wins by 4 points at a twelfth of the price, $0.30 per task against $3.61. Gemma 4's best harness beats GLM-5.2's four worst. A 26B model in the right scaffold is not far off a 744B model in the wrong one. Cost per solved task: $0.84 for Gemma 4 + crush at 36%. The cheapest GLM-5.2 setup that scores as well is openclaw at 38%, for $7.05. Output tokens per task span 16k to 621k across harnesses: a 39x spread in what you pay for, buying a 2x spread in what you get. 97% of input tokens are re-sent conversation prefix, so prompt caching is super important. The setup: every harness run against both models on the same 250 SWE-bench Pro tasks, one rollout each, priced at list API rates for the tokens each one actually spent. In the plot, a dark ring means the pairing sits on the Pareto frontier across both models and a faded dot means you could score higher for less elsewhere; 2 of the 10 harnesses (goose and hermes) are left out of the figure for legibility