Harness Engineering Practices P10. Two-Phase Design — Read-Only Exploration, Then Implementation 🎯 Point "Understand before you write" is a human virtue. Enforce the same discipline on agents — not with wishes, but with permissions. 📝 Overview Force a write-locked exploration phase first, requiring a plan before entering the implementation phase. Structurally prevent premature editing to improve plan quality. Phase boundaries are enforced by the harness through permissions, not by prompting. 🔍 Explanation Agents tend to "just start writing." They edit files before grasping the full picture, then realize "the approach was wrong to begin with," causing massive rework. Two-phase design permits only file reads, searches, and symbol resolution in the first phase, disabling edit tools. The agent focuses solely on exploring, understanding, and planning. Edit permissions unlock only after the plan is approved. This structurally eliminates the risk of "impulsive edits that break things." 🛠 How to Practice - In Phase 1, disable edit tools and configure a tool set permitting only file reads, searches, and symbol resolution - Require a plan file (plan.md) as Phase 1 output and make plan approval the gate for entering Phase 2 - Enforce phase boundaries at the tool permission level, not through prompt-based requests - Set time and step limits on the exploration phase to prevent context exhaustion 💼 Use Cases - Issue-to-PR agents: analyze the issue and draft a plan in read-only mode, implement only after plan approval - Legacy code modernization: map and understand modules first, then apply changes - Incident response: separate diagnosis (read-only, safe, autonomous) from remediation (write, gated) ⚠ Pitfalls An overly long exploration phase can exhaust the context window. Knowledge gathered during exploration may also become stale by implementation time (combine with P3's TTL). "Asking nicely in the prompt" isn't sufficient for phase boundaries — enforce them at the tool permission level. #HarnessEngineering# #AIAgent#

Harness optimization is getting real receipts. AutoSaddler is offline harness learning from agent failure traces. Not another prompt tweak loop. It patches prompts, tools, and middleware as code. Then it keeps updates that survive a held out set. On the test sets (Pass@1): GAIA2: 53.0 → 62.0 (+9.0) SWE-Bench Pro: 37.3 → 46.9 (+9.6) Terminal-Bench 2.0: 40.0 → 50.0 (+10.0) That TB2 number also clears the expert tuned Terminus KIRA at 47.5. Kill generalization aware selection and GAIA2 falls to 50.6, under the default agent. Deep diagnosis and structured patches help. Dev set filtering is what stops the harness from overfitting the mini batch. On GAIA2, Figure 1b, about 147 leveraged traces to the best dev score vs about 1,400 for Meta-Harness.

Harness Engineering Anti-Patterns AP8. Gate Fatigue / Rubber-Stamping 🎯 Point Approval dialogs even for reversible operations, dozens of reflexive "approve" clicks per day. Excessive gates neutralize the gates themselves. ❗ Problem Too many approval requests cause humans to reflexively skip gates. Through the same mechanism as alert fatigue, even truly dangerous irreversible action approvals get rubber-stamped, completely defeating the purpose of gates. 🔍 Mechanism & Symptoms "Human approval on everything" looks safe and reassures auditors, making this anti-pattern organizationally popular. But meaningless approval repetition makes approval reflexive — identical to alert fatigue. If you're asked "may I edit this file?" 50 times a day, the 51st "may I delete the production DB?" gets the same reflexive approval. Symptoms: approval time under 1 second (proof of not reading), all approvals logged as "reviewed" but actually unreviewed, important operation approvals buried among trivial ones, and team complaints about "annoying approvals." 📋 Scenarios - The agent requests approval for every file read. Developers habitually select "allow all," and later reflexively approve when the agent executes a force-push. - Security team mandates "approval required for all operations." Developers process 200 approvals daily and create a script to batch-approve everything. Gates are completely hollow. - In pair programming, every agent edit requires diff preview and approval. Developers mash Enter without reading diffs, and unintended changes slip through. 🛡 How to Avoid - Design permissions around "revocability" not "capability" — freely permit reversible operations - Consolidate only irreversible operations into "heavyweight ceremony" gates, preserving gate scarcity - Measure approval response times — sub-1-second consistently indicates gate decay - Consciously minimize gate count, limiting them to "moments that truly require human judgment" #HarnessEngineering# #AIAgent#

Harness Engineering Practices P7. Negative Verification — Regression and Blast Radius Gates 🎯 Point Agents optimize for "my change works" and underweight "I haven't broken anything else." Positive verification alone won't catch escaped defects. 📝 Overview Add blast radius checks to completion gates — beyond running the full existing test suite, verify "who imports the symbols I touched." Negative verification confirms not just that your code works, but that nothing else is broken. 🔍 Explanation Agents naturally focus on "tests related to my change pass." But change impact doesn't stop at the changed code. Altering a function signature breaks callers; changing shared module behavior affects all dependents. Negative verification is the explicit mechanism for verifying "nothing was broken." Identify import sites of changed symbols, run their tests too. Embedding impact visualization and test execution into the completion gate structurally prevents escaped defects. 🛠 How to Practice - Integrate tooling into completion gates that auto-identifies import sites of changed symbols via static analysis (import analysis, call graphs) - Automatically add identified dependent tests to the execution set alongside the existing test suite - Visualize change blast radius (impacted file count, module count) alongside the diff and present it to reviewers - Combine coverage data with static analysis to identify and flag under-tested impact areas 💼 Use Cases - Issue-to-PR agents: auto-run all dependent tests when shared utilities are modified - Migration agents: verify the full impact zone of API signature changes - Legacy code modernization: cover change ripple effects with characterization tests ⚠ Pitfalls Exhaustive blast radius checking can spiral into running the entire monorepo test suite, which is impractical. Combining static analysis (import analysis, call graphs) with dynamic analysis (coverage data) is most effective. Also, negative verification existing doesn't mean you can neglect positive verification — both are necessary. #HarnessEngineering# #QualityAssurance#

Harness Engineering Anti-Patterns AP7. The Misplaced Determinism Boundary 🎯 Point Trusting the LLM's goodwill to run tests, while cramming edge case judgment into rigid rules. Get the boundary wrong and you lose both reliability and adaptability. ❗ Problem Putting probabilistic elements where determinism is needed leaks reliability; putting deterministic rules where judgment is needed loses adaptability. The result: instability on simple tasks, rigidity on complex ones, or both simultaneously. 🔍 Mechanism & Symptoms This misplacement takes two forms. Form A: cramming LLM-judgment long tails into rigid rules. Rules feel predictable, but break brittly on edge cases. Form B: entrusting deterministic operations (test execution, gate decisions, retries) to LLM goodwill. Delegating to the model feels easier, but produces instability — tests forgotten 1 in 100 runs. Symptoms of Form A: "rules break on unexpected cases," "need new rules for every new pattern." Form B: "test execution forgotten," "gates skipped," "randomly stops working." 📋 Scenarios - Form A: A rigid rule "import changes must be at file top" is set. When circular import resolution requires otherwise, the rule blocks the agent and it gets stuck. - Form B: "Always run tests" is stated in the prompt but not harness-enforced. The agent runs tests 95% of the time but declares completion without tests the other 5%. - Both: Most codemod work could be deterministic AST transforms, but everything is delegated to the LLM (Form B). Meanwhile, edge cases needing LLM judgment get rigid "skip in this case" rules (Form A). Both are wrong. 🛡 How to Avoid - Classify all harness operations as "requires judgment" vs. "can execute deterministically" and make the boundary explicit - Enforce test execution, lint, build, and gate decisions in deterministic code — don't "ask nicely" - Restrict LLM use to genuinely judgment-dependent parts (root cause analysis, strategy decisions, code generation) - Periodically review and adjust the boundary as models evolve #HarnessEngineering# #AIAgent#

Harness Engineering Practices P20. Make Legibility and Calibrated Uncertainty an SLO 🎯 Point Now that generation is cheap, the real bottleneck is "human review time." Diffs should be optimized not just for correctness, but for reviewability. 📝 Overview Optimize diffs for review time, not just correctness. Small, focused PRs. Explanations that tell "why." Explicit flagging of risky areas. Additionally, have the agent explicitly output "areas of low confidence" so the harness can route them to additional verification or human review. Calibrated uncertainty is more valuable than false confidence. 🔍 Explanation As agent generation speed increases, the bottleneck shifts from "writing code" to "reviewing code." Giant PRs, unexplained changes, confidently-presented but actually uncertain implementations — these explosively consume reviewer time. Treating legibility as an SLO (Service Level Objective) and measuring/optimizing PR size, explanation presence, and change rationale improves overall throughput. Having agents explicitly state "I'm not confident here" and "this needs human verification" lets reviewers focus on what matters. This is the most overlooked practice for building trustworthy autonomous agents. 🛠 How to Practice - Add "change rationale," "confidence level (high/medium/low)," and "review focus areas" fields to PR templates and require the agent to fill them - Set PR size limits and force splitting when exceeded - Require "low confidence" markers in agent output so the harness can route those areas to additional verification - Measure review time per PR and identify causes of long reviews (giant diffs, missing explanations, etc.) for improvement 💼 Use Cases - Issue-to-PR agents including change rationale and confidence markers in PRs - Code review agents suppressing low-confidence trivial comments and focusing on types humans miss - Migration: keeping per-unit PRs small and focused to distribute reviewer load ⚠ Pitfalls Over-optimizing for legibility can make agent output overly conservative. "Uncertainty expression" can also become noise — an agent that says "I'm not confident" about everything is useless. Calibration is key: accurately marking only genuinely uncertain areas is what creates value. Don't forget to measure review time either. Quantitatively tracking whether PR spam or giant diffs are crushing review bandwidth is the starting point for improvement. #HarnessEngineering# #CodeReview#

Harness Engineering Practices P19. Turn Failures into Data Assets (The Harness's Own Retro Loop) 🎯 Point A harness that repeats the same failures is an unimprovable black box. Log failures and feed them back into harness improvement. 📝 Overview Log every human intervention, rollback, and escaped defect with its cause, then feed this into harness improvements (new gates, new instructions, new tools). The harness should have CI for itself. This is what separates maturity L3 (measuring) from L4 (continuously improving). 🔍 Explanation When agents fail, most organizations conclude "the model is bad." But the real question is "why couldn't the harness prevent this failure?" A human intervention means the harness lacked a guardrail or verification. A needed rollback means the circuit breaker didn't trigger. An escaped defect means the verifier was insufficient. Recording these events with root cause analysis and converting them into harness improvement actions (adding gates, updating instruction files, improving tools) is the loop that matures a harness into a product. 🛠 How to Practice - Record all human interventions, rollbacks, and escaped defects in structured logs with cause classification - Run regular retrospectives (weekly or biweekly) to identify recurring failure patterns - For each failure pattern, select and implement the most effective improvement action (new gate, instruction addition, tool improvement) - Measure improvement action effectiveness and retract low-impact ones to try different approaches 💼 Use Cases - Weekly analysis of issue-to-PR agent failures to identify harness improvement points - CI auto-maintenance: tracking false positive causes to improve triage logic - Incident response: deriving observability access improvements from cases where agent recommendations were inaccurate ⚠ Pitfalls Adding a rule after every failure leads to "Scaffolding Ratchet" (AP3). Turning failures into data assets isn't about adding more rules — it's about root cause analysis and choosing the most effective improvement. Logging without analysis accumulates data without generating value. Regular retrospective processes are essential. #HarnessEngineering# #ContinuousImprovement#

Harness Engineering Anti-Patterns AP10. The Unobservable Black Box 🎯 Point Was it the model, the prompt, the tool, the context, or the environment? Nobody can tell. A harness where improvement runs on superstition and intuition is a harness that cannot improve. ❗ Problem Failures can't be attributed to subsystems, so no one knows what to fix. Improvement becomes superstition and guesswork, and the harness's retrospective loop stops turning. Combined with "metric monoculture" — tracking only a single metric — unmeasured qualities silently degrade. 🔍 Mechanism & Symptoms Observability is unglamorous infrastructure work, and agents "mostly work," so this anti-pattern gets deprioritized. But without the ability to attribute failures to subsystems, the retrospective loop can't run and the harness becomes unimprovable. Further, tracking a single metric (e.g., success rate only) creates "metric monoculture" where unmeasured virtues (review time, regression rate, code maintainability) quietly suffer. Symptoms: "why did it fail? no idea" is frequent, improvement efforts default to "tweak the prompt," model vs. harness issues are indistinguishable, and success rate improves while reviewer frustration grows. 📋 Scenarios - An agent fails a task but no one can determine whether it was model reasoning error, insufficient context, a tool bug, or an environment issue. The team repeats "let's make the prompt more detailed" as symptomatic treatment. - Success rate is tracked as the sole metric. It improves to 80%, but no one notices that review time for successful cases has tripled. - A model upgrade shows no performance change. Whether it's a model issue or harness scaffolding constraining performance (AP3) can't be distinguished, and investment decisions become superstition. 🛡 How to Avoid - Trace decisions, tool calls, and context transitions, making them attributable to the 7 subsystems (perception, action, feedback, control, memory, guardrails, interface) - Measure in bundles, not single metrics (success rate, intervention rate, rework rate, regression rate, cost, review time, confidence calibration) - A/B test harness changes with the model held fixed to attribute improvements to the harness - Invest in harness observability as "unglamorous but essential infrastructure" and build it as the foundation for improvement loops #HarnessEngineering# #AIAgent#

Harness Engineering Practices P17. Dry Run by Default and Blast Radius Preview 🎯 Point "Restarting 40 pods" should be presented before execution, not discovered after. Preview side effects before they happen. 📝 Overview Actions with side effects first show a preview of their effects (which files, which lines, which pods), then pass through a gate or human confirmation before execution. Default to dry run (show results without executing), and execute only after explicit approval. 🔍 Explanation Knowing "what the agent will do" in advance is foundational to safety. Diff previews, lists of affected services, pod restart counts, message contents — presenting these before execution enables informed human judgment. In incident response especially, presenting remediation blast radius and rollback plans together, then executing with monitoring post-approval, is an effective pattern. Making dry run the default structurally eliminates "accidental execution" risks. 🛠 How to Practice - Implement "preview mode" for all side-effecting operations, displaying impact scope before execution - Include quantitative info in dry run output: number of affected files, pods, changed lines, etc. - Make dry run mandatory for irreversible operations; keep it optional for reversible ones to maintain efficiency - Record dry run vs. actual execution discrepancies and continuously improve dry run accuracy 💼 Use Cases - Incident response: present blast radius and rollback plan for remediation actions upfront - Pair programming: display agent edits as diff previews before applying - Migration: canary-apply changes to a subset first and verify before full rollout ⚠ Pitfalls Dry runs aren't always accurate — environmental differences can cause failures at execution time that didn't appear in dry run. If dry runs become reflexively skipped, they lose meaning. Combine with P15 (revocability): execute reversible operations without dry run, require dry run only for irreversible operations. #HarnessEngineering# #AIAgent#