Register and share your invite link to earn from video plays and referrals.

VulcanBench
@VulcanBench
Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.
Joined March 2020
63 Following    2.3K Followers
Really interesting experiment.
Same weights. $50 of harness search. 3x on Terminal-Bench. We optimized the OpenCode harness for GPT-OSS-20B (high reasoning) with a meta-harness: an agent that reads failures, patches the harness, and validates the improvement. The model never changed. On held-out Terminal-Bench 2.1, the optimized harness scored 14.8%, up from OpenCode's 4.8%. That is 3.08x, from $49.97 of API credits. Three harness fixes carried most of the gain: - i7, verify before stopping: the model often edited code and ended the session without testing it. The harness gives it another turn to run the code. - i14, continue announced actions: the model would say "now run X" and then stop. The harness keeps the session going so it runs X. - i23, repair malformed tool calls: one extra `]` made OpenCode reject otherwise valid commands. The harness repairs that clear-cut error. How the meta-harness ran: 1. Start: stock OpenCode v1.18.13 + GPT-OSS-20B scored 8.5% on the dev split. 2. Iterate: Claude Code (Opus 5 max) reads the failed dev trajectories and proposes one patch. We build it and run it once per dev task, paired against the current parent. 3. Validate: candidates that clear a pre-registered margin get a 5-trial validation. They become the new parent only if their validated score surpasses the current one. 4. Stop: the loop ended after 23 iterations and about 2.4k dev attempts, when the $50 search budget was spent. Takeaway: pre-training puts capability in the weights, and post-training makes that capability usable. Harness optimization continues the same work at inference time, turning the model's behavior into more completed tasks. Same weights, 3x the finished tasks.
Show more