Harness optimization is getting real receipts.
AutoSaddler is offline harness learning from agent failure traces. Not another prompt tweak loop.
It patches prompts, tools, and middleware as code. Then it keeps updates that survive a held out set.
On the test sets (Pass
@1):
GAIA2: 53.0 → 62.0 (+9.0)
SWE-Bench Pro: 37.3 → 46.9 (+9.6)
Terminal-Bench 2.0: 40.0 → 50.0 (+10.0)
That TB2 number also clears the expert tuned Terminus KIRA at 47.5.
Kill generalization aware selection and GAIA2 falls to 50.6, under the default agent. Deep diagnosis and structured patches help. Dev set filtering is what stops the harness from overfitting the mini batch.
On GAIA2, Figure 1b, about 147 leveraged traces to the best dev score vs about 1,400 for Meta-Harness.