Harness Engineering Practices P17. Dry Run by Default and Blast Radius Preview 🎯 Point "Restarting 40 pods" should be presented before execution, not discovered after. Preview side effects before they happen. 📝 Overview Actions with side effects first show a preview of their effects (which files, which lines, which pods), then pass through a gate or human confirmation before execution. Default to dry run (show results without executing), and execute only after explicit approval. 🔍 Explanation Knowing "what the agent will do" in advance is foundational to safety. Diff previews, lists of affected services, pod restart counts, message contents — presenting these before execution enables informed human judgment. In incident response especially, presenting remediation blast radius and rollback plans together, then executing with monitoring post-approval, is an effective pattern. Making dry run the default structurally eliminates "accidental execution" risks. 🛠 How to Practice - Implement "preview mode" for all side-effecting operations, displaying impact scope before execution - Include quantitative info in dry run output: number of affected files, pods, changed lines, etc. - Make dry run mandatory for irreversible operations; keep it optional for reversible ones to maintain efficiency - Record dry run vs. actual execution discrepancies and continuously improve dry run accuracy 💼 Use Cases - Incident response: present blast radius and rollback plan for remediation actions upfront - Pair programming: display agent edits as diff previews before applying - Migration: canary-apply changes to a subset first and verify before full rollout ⚠ Pitfalls Dry runs aren't always accurate — environmental differences can cause failures at execution time that didn't appear in dry run. If dry runs become reflexively skipped, they lose meaning. Combine with P15 (revocability): execute reversible operations without dry run, require dry run only for irreversible operations. #HarnessEngineering# #AIAgent#

Harness Engineering Practices P16. Trust Through Earned Credentials (Capability Ramping) 🎯 Point No company gives a new hire production deploy access on day one. Agents need a system for "earning trust through track record" too. 📝 Overview New harnesses start with strict gates. When an agent builds a track record on a task type (high success rate, zero regressions), autonomy expands for that type only. Trust is earned per task type based on measurement, and is always revocable. 🔍 Explanation Granting full autonomy for all tasks from the start is dangerous, but maintaining strict gates forever is inefficient. The solution is gradual trust earning. For example, after 10 consecutive successes with zero regressions on lint fixes, permit auto-merge for lint fixes only. Security patches stay gated until trust is earned on a separate track. The key principles: trust is independent per task type, based on measurement, and always retractable. When a regression occurs, immediately downgrade trust for that task type. 🛠 How to Practice - Define trust levels per task type (e.g., L0 = all approvals required, L1 = light autonomy, L2 = auto-merge allowed) - Define promotion criteria numerically (e.g., 95%+ success rate over last 20 tasks with zero regressions) - Define demotion criteria too (immediately drop to L0 on regression) - Log trust level changes and their rationale to ensure decision transparency 💼 Use Cases - CI auto-maintenance: gradually permit auto-merge for low-risk types (lint patches) only - Issue-to-PR agents: adjust human review granularity based on bug fix success track record - Legacy code modernization: characterization test creation is autonomous; code changes proceed cautiously with canary deploys ⚠ Pitfalls Unclear trust-earning criteria lead to either permanent strict gates or unnoticed drift toward laxity. Define criteria numerically (success rate above X%, zero regressions in last N tasks) with automated evaluation. Also, "high success rate" doesn't equal "safe" — consider whether the agent is padding stats with easy tasks. #HarnessEngineering# #AIAgent#

Harness Engineering Practices P14. Independent Self-Critique Pass (Separate Generator and Critic Context) 🎯 Point Do you trust an agent that reviews its own code and says "looks good"? Reviews only function when separated from the author. 📝 Overview After implementation, critique the diff against the spec in a separate context (ideally with a different role prompt). To avoid the "I just wrote it so it must be correct" bias, the generator and critic must not share context. 🔍 Explanation For the same reason human code reviews are performed by someone other than the author, agent self-critique requires context separation. When generation and critique happen in the same context, the "why I wrote it this way" bias carries directly into critique, making problems easier to miss. In a separate context — ideally a different prompt or model instance — showing only the diff and spec enables a fresh-perspective review. This adds computational cost, but the defect-prevention payoff is significant. 🛠 How to Practice - After implementation, launch a separate context (new conversation session) and provide only the diff and spec for review - Give critics focused roles ("spec divergence," "security risks," "regression potential," etc.) - Where possible, use different models or prompts for generation and critique to ensure perspective diversity - Have critique results output in structured format so the harness can auto-decide actions (send fix instructions, report to humans) 💼 Use Cases - Issue-to-PR agents running a self-review with a different prompt before completion - Autonomous bug fixing: validating fix soundness in a separate context - Code generation: running security-focused reviews as a separate agent ⚠ Pitfalls A critic designed to "flag everything" produces a flood of trivial comments, creating infinite implement-fix-critique-fix loops. Give critics focused roles: "spec divergence," "security risks," "regression potential." Low-quality critics also risk unnecessarily modifying correct implementations. #HarnessEngineering# #CodeReview#

Harness Engineering Practices P12. Semantic Circuit Breakers and "Discard Context" Restarts 🎯 Point Ever seen an agent repeat the same error three times? "Push through" is far slower than "restart with a lesson learned." 📝 Overview Instead of simple max-iteration limits, semantically detect "same error twice," "oscillating between two edits," or "no net progress" and halt. When stuck, don't keep kneading in contaminated context. Roll back to the last good checkpoint, discard failure history from context, carry just a one-line lesson, and restart with a fresh approach. 🔍 Explanation When an agent gets stuck, the least efficient response is "try harder in the same context." As failure history accumulates in context, the agent gets dragged toward repeating the same mistakes. Semantic circuit breakers judge "is there actual progress" rather than counting iterations. When they detect oscillation (change A to B then back to A) or repeated identical errors, they trigger a git rollback to the last clean state, then restart in fresh context with only "last time X failed, try a different approach." Git is the agent's Undo. 🛠 How to Practice - Implement logic in the harness that compares error messages and test results to detect "same error twice" and "A-B edit oscillation" - On halt, git checkout to the last clean commit and discard contaminated context - On restart, carry only a one-line lesson ("last time X failed") into the fresh context - Tune circuit breaker thresholds per task difficulty (strict for easy tasks, more lenient for hard ones) 💼 Use Cases - Issue-to-PR agents looping on the same test failure they can't fix - Compile error fixing: detecting fix-then-break-then-revert cycles - Legacy code modernization: recognizing when a fundamentally different approach is needed ⚠ Pitfalls Too-sensitive circuit breakers cut off agents one step from a solution. Too-lenient ones waste cost in infinite loops. Tuning the definition of "progress" per task is critical. Distilling accurate lessons at restart time is also hard — wrong lessons carried into a restart can trigger different failure patterns. #HarnessEngineering# #AIAgent#

Harness Engineering Practices P10. Two-Phase Design — Read-Only Exploration, Then Implementation 🎯 Point "Understand before you write" is a human virtue. Enforce the same discipline on agents — not with wishes, but with permissions. 📝 Overview Force a write-locked exploration phase first, requiring a plan before entering the implementation phase. Structurally prevent premature editing to improve plan quality. Phase boundaries are enforced by the harness through permissions, not by prompting. 🔍 Explanation Agents tend to "just start writing." They edit files before grasping the full picture, then realize "the approach was wrong to begin with," causing massive rework. Two-phase design permits only file reads, searches, and symbol resolution in the first phase, disabling edit tools. The agent focuses solely on exploring, understanding, and planning. Edit permissions unlock only after the plan is approved. This structurally eliminates the risk of "impulsive edits that break things." 🛠 How to Practice - In Phase 1, disable edit tools and configure a tool set permitting only file reads, searches, and symbol resolution - Require a plan file (plan.md) as Phase 1 output and make plan approval the gate for entering Phase 2 - Enforce phase boundaries at the tool permission level, not through prompt-based requests - Set time and step limits on the exploration phase to prevent context exhaustion 💼 Use Cases - Issue-to-PR agents: analyze the issue and draft a plan in read-only mode, implement only after plan approval - Legacy code modernization: map and understand modules first, then apply changes - Incident response: separate diagnosis (read-only, safe, autonomous) from remediation (write, gated) ⚠ Pitfalls An overly long exploration phase can exhaust the context window. Knowledge gathered during exploration may also become stale by implementation time (combine with P3's TTL). "Asking nicely in the prompt" isn't sufficient for phase boundaries — enforce them at the tool permission level. #HarnessEngineering# #AIAgent#

Harness Engineering Anti-Patterns AP8. Gate Fatigue / Rubber-Stamping 🎯 Point Approval dialogs even for reversible operations, dozens of reflexive "approve" clicks per day. Excessive gates neutralize the gates themselves. ❗ Problem Too many approval requests cause humans to reflexively skip gates. Through the same mechanism as alert fatigue, even truly dangerous irreversible action approvals get rubber-stamped, completely defeating the purpose of gates. 🔍 Mechanism & Symptoms "Human approval on everything" looks safe and reassures auditors, making this anti-pattern organizationally popular. But meaningless approval repetition makes approval reflexive — identical to alert fatigue. If you're asked "may I edit this file?" 50 times a day, the 51st "may I delete the production DB?" gets the same reflexive approval. Symptoms: approval time under 1 second (proof of not reading), all approvals logged as "reviewed" but actually unreviewed, important operation approvals buried among trivial ones, and team complaints about "annoying approvals." 📋 Scenarios - The agent requests approval for every file read. Developers habitually select "allow all," and later reflexively approve when the agent executes a force-push. - Security team mandates "approval required for all operations." Developers process 200 approvals daily and create a script to batch-approve everything. Gates are completely hollow. - In pair programming, every agent edit requires diff preview and approval. Developers mash Enter without reading diffs, and unintended changes slip through. 🛡 How to Avoid - Design permissions around "revocability" not "capability" — freely permit reversible operations - Consolidate only irreversible operations into "heavyweight ceremony" gates, preserving gate scarcity - Measure approval response times — sub-1-second consistently indicates gate decay - Consciously minimize gate count, limiting them to "moments that truly require human judgment" #HarnessEngineering# #AIAgent#

Harness Engineering Practices P7. Negative Verification — Regression and Blast Radius Gates 🎯 Point Agents optimize for "my change works" and underweight "I haven't broken anything else." Positive verification alone won't catch escaped defects. 📝 Overview Add blast radius checks to completion gates — beyond running the full existing test suite, verify "who imports the symbols I touched." Negative verification confirms not just that your code works, but that nothing else is broken. 🔍 Explanation Agents naturally focus on "tests related to my change pass." But change impact doesn't stop at the changed code. Altering a function signature breaks callers; changing shared module behavior affects all dependents. Negative verification is the explicit mechanism for verifying "nothing was broken." Identify import sites of changed symbols, run their tests too. Embedding impact visualization and test execution into the completion gate structurally prevents escaped defects. 🛠 How to Practice - Integrate tooling into completion gates that auto-identifies import sites of changed symbols via static analysis (import analysis, call graphs) - Automatically add identified dependent tests to the execution set alongside the existing test suite - Visualize change blast radius (impacted file count, module count) alongside the diff and present it to reviewers - Combine coverage data with static analysis to identify and flag under-tested impact areas 💼 Use Cases - Issue-to-PR agents: auto-run all dependent tests when shared utilities are modified - Migration agents: verify the full impact zone of API signature changes - Legacy code modernization: cover change ripple effects with characterization tests ⚠ Pitfalls Exhaustive blast radius checking can spiral into running the entire monorepo test suite, which is impractical. Combining static analysis (import analysis, call graphs) with dynamic analysis (coverage data) is most effective. Also, negative verification existing doesn't mean you can neglect positive verification — both are necessary. #HarnessEngineering# #QualityAssurance#

Harness Engineering Anti-Patterns AP7. The Misplaced Determinism Boundary 🎯 Point Trusting the LLM's goodwill to run tests, while cramming edge case judgment into rigid rules. Get the boundary wrong and you lose both reliability and adaptability. ❗ Problem Putting probabilistic elements where determinism is needed leaks reliability; putting deterministic rules where judgment is needed loses adaptability. The result: instability on simple tasks, rigidity on complex ones, or both simultaneously. 🔍 Mechanism & Symptoms This misplacement takes two forms. Form A: cramming LLM-judgment long tails into rigid rules. Rules feel predictable, but break brittly on edge cases. Form B: entrusting deterministic operations (test execution, gate decisions, retries) to LLM goodwill. Delegating to the model feels easier, but produces instability — tests forgotten 1 in 100 runs. Symptoms of Form A: "rules break on unexpected cases," "need new rules for every new pattern." Form B: "test execution forgotten," "gates skipped," "randomly stops working." 📋 Scenarios - Form A: A rigid rule "import changes must be at file top" is set. When circular import resolution requires otherwise, the rule blocks the agent and it gets stuck. - Form B: "Always run tests" is stated in the prompt but not harness-enforced. The agent runs tests 95% of the time but declares completion without tests the other 5%. - Both: Most codemod work could be deterministic AST transforms, but everything is delegated to the LLM (Form B). Meanwhile, edge cases needing LLM judgment get rigid "skip in this case" rules (Form A). Both are wrong. 🛡 How to Avoid - Classify all harness operations as "requires judgment" vs. "can execute deterministically" and make the boundary explicit - Enforce test execution, lint, build, and gate decisions in deterministic code — don't "ask nicely" - Restrict LLM use to genuinely judgment-dependent parts (root cause analysis, strategy decisions, code generation) - Periodically review and adjust the boundary as models evolve #HarnessEngineering# #AIAgent#

Harness Engineering Practices P20. Make Legibility and Calibrated Uncertainty an SLO 🎯 Point Now that generation is cheap, the real bottleneck is "human review time." Diffs should be optimized not just for correctness, but for reviewability. 📝 Overview Optimize diffs for review time, not just correctness. Small, focused PRs. Explanations that tell "why." Explicit flagging of risky areas. Additionally, have the agent explicitly output "areas of low confidence" so the harness can route them to additional verification or human review. Calibrated uncertainty is more valuable than false confidence. 🔍 Explanation As agent generation speed increases, the bottleneck shifts from "writing code" to "reviewing code." Giant PRs, unexplained changes, confidently-presented but actually uncertain implementations — these explosively consume reviewer time. Treating legibility as an SLO (Service Level Objective) and measuring/optimizing PR size, explanation presence, and change rationale improves overall throughput. Having agents explicitly state "I'm not confident here" and "this needs human verification" lets reviewers focus on what matters. This is the most overlooked practice for building trustworthy autonomous agents. 🛠 How to Practice - Add "change rationale," "confidence level (high/medium/low)," and "review focus areas" fields to PR templates and require the agent to fill them - Set PR size limits and force splitting when exceeded - Require "low confidence" markers in agent output so the harness can route those areas to additional verification - Measure review time per PR and identify causes of long reviews (giant diffs, missing explanations, etc.) for improvement 💼 Use Cases - Issue-to-PR agents including change rationale and confidence markers in PRs - Code review agents suppressing low-confidence trivial comments and focusing on types humans miss - Migration: keeping per-unit PRs small and focused to distribute reviewer load ⚠ Pitfalls Over-optimizing for legibility can make agent output overly conservative. "Uncertainty expression" can also become noise — an agent that says "I'm not confident" about everything is useless. Calibration is key: accurately marking only genuinely uncertain areas is what creates value. Don't forget to measure review time either. Quantitatively tracking whether PR spam or giant diffs are crushing review bandwidth is the starting point for improvement. #HarnessEngineering# #CodeReview#

Harness Engineering Practices P19. Turn Failures into Data Assets (The Harness's Own Retro Loop) 🎯 Point A harness that repeats the same failures is an unimprovable black box. Log failures and feed them back into harness improvement. 📝 Overview Log every human intervention, rollback, and escaped defect with its cause, then feed this into harness improvements (new gates, new instructions, new tools). The harness should have CI for itself. This is what separates maturity L3 (measuring) from L4 (continuously improving). 🔍 Explanation When agents fail, most organizations conclude "the model is bad." But the real question is "why couldn't the harness prevent this failure?" A human intervention means the harness lacked a guardrail or verification. A needed rollback means the circuit breaker didn't trigger. An escaped defect means the verifier was insufficient. Recording these events with root cause analysis and converting them into harness improvement actions (adding gates, updating instruction files, improving tools) is the loop that matures a harness into a product. 🛠 How to Practice - Record all human interventions, rollbacks, and escaped defects in structured logs with cause classification - Run regular retrospectives (weekly or biweekly) to identify recurring failure patterns - For each failure pattern, select and implement the most effective improvement action (new gate, instruction addition, tool improvement) - Measure improvement action effectiveness and retract low-impact ones to try different approaches 💼 Use Cases - Weekly analysis of issue-to-PR agent failures to identify harness improvement points - CI auto-maintenance: tracking false positive causes to improve triage logic - Incident response: deriving observability access improvements from cases where agent recommendations were inaccurate ⚠ Pitfalls Adding a rule after every failure leads to "Scaffolding Ratchet" (AP3). Turning failures into data assets isn't about adding more rules — it's about root cause analysis and choosing the most effective improvement. Logging without analysis accumulates data without generating value. Regular retrospective processes are essential. #HarnessEngineering# #ContinuousImprovement#