Harness Engineering Practices P19. Turn Failures into Data Assets (The Harness's Own Retro Loop) 🎯 Point A harness that repeats the same failures is an unimprovable black box. Log failures and feed them back into harness improvement. 📝 Overview Log every human intervention, rollback, and escaped defect with its cause, then feed this into harness improvements (new gates, new instructions, new tools). The harness should have CI for itself. This is what separates maturity L3 (measuring) from L4 (continuously improving). 🔍 Explanation When agents fail, most organizations conclude "the model is bad." But the real question is "why couldn't the harness prevent this failure?" A human intervention means the harness lacked a guardrail or verification. A needed rollback means the circuit breaker didn't trigger. An escaped defect means the verifier was insufficient. Recording these events with root cause analysis and converting them into harness improvement actions (adding gates, updating instruction files, improving tools) is the loop that matures a harness into a product. 🛠 How to Practice - Record all human interventions, rollbacks, and escaped defects in structured logs with cause classification - Run regular retrospectives (weekly or biweekly) to identify recurring failure patterns - For each failure pattern, select and implement the most effective improvement action (new gate, instruction addition, tool improvement) - Measure improvement action effectiveness and retract low-impact ones to try different approaches 💼 Use Cases - Weekly analysis of issue-to-PR agent failures to identify harness improvement points - CI auto-maintenance: tracking false positive causes to improve triage logic - Incident response: deriving observability access improvements from cases where agent recommendations were inaccurate ⚠ Pitfalls Adding a rule after every failure leads to "Scaffolding Ratchet" (AP3). Turning failures into data assets isn't about adding more rules — it's about root cause analysis and choosing the most effective improvement. Logging without analysis accumulates data without generating value. Regular retrospective processes are essential. #HarnessEngineering# #ContinuousImprovement#