Harness Engineering Practices P7. Negative Verification — Regression and Blast Radius Gates 🎯 Point Agents optimize for "my change works" and underweight "I haven't broken anything else." Positive verification alone won't catch escaped defects. 📝 Overview Add blast radius checks to completion gates — beyond running the full existing test suite, verify "who imports the symbols I touched." Negative verification confirms not just that your code works, but that nothing else is broken. 🔍 Explanation Agents naturally focus on "tests related to my change pass." But change impact doesn't stop at the changed code. Altering a function signature breaks callers; changing shared module behavior affects all dependents. Negative verification is the explicit mechanism for verifying "nothing was broken." Identify import sites of changed symbols, run their tests too. Embedding impact visualization and test execution into the completion gate structurally prevents escaped defects. 🛠 How to Practice - Integrate tooling into completion gates that auto-identifies import sites of changed symbols via static analysis (import analysis, call graphs) - Automatically add identified dependent tests to the execution set alongside the existing test suite - Visualize change blast radius (impacted file count, module count) alongside the diff and present it to reviewers - Combine coverage data with static analysis to identify and flag under-tested impact areas 💼 Use Cases - Issue-to-PR agents: auto-run all dependent tests when shared utilities are modified - Migration agents: verify the full impact zone of API signature changes - Legacy code modernization: cover change ripple effects with characterization tests ⚠ Pitfalls Exhaustive blast radius checking can spiral into running the entire monorepo test suite, which is impractical. Combining static analysis (import analysis, call graphs) with dynamic analysis (coverage data) is most effective. Also, negative verification existing doesn't mean you can neglect positive verification — both are necessary. #HarnessEngineering# #QualityAssurance#

Harness Engineering Anti-Patterns AP7. The Misplaced Determinism Boundary 🎯 Point Trusting the LLM's goodwill to run tests, while cramming edge case judgment into rigid rules. Get the boundary wrong and you lose both reliability and adaptability. ❗ Problem Putting probabilistic elements where determinism is needed leaks reliability; putting deterministic rules where judgment is needed loses adaptability. The result: instability on simple tasks, rigidity on complex ones, or both simultaneously. 🔍 Mechanism & Symptoms This misplacement takes two forms. Form A: cramming LLM-judgment long tails into rigid rules. Rules feel predictable, but break brittly on edge cases. Form B: entrusting deterministic operations (test execution, gate decisions, retries) to LLM goodwill. Delegating to the model feels easier, but produces instability — tests forgotten 1 in 100 runs. Symptoms of Form A: "rules break on unexpected cases," "need new rules for every new pattern." Form B: "test execution forgotten," "gates skipped," "randomly stops working." 📋 Scenarios - Form A: A rigid rule "import changes must be at file top" is set. When circular import resolution requires otherwise, the rule blocks the agent and it gets stuck. - Form B: "Always run tests" is stated in the prompt but not harness-enforced. The agent runs tests 95% of the time but declares completion without tests the other 5%. - Both: Most codemod work could be deterministic AST transforms, but everything is delegated to the LLM (Form B). Meanwhile, edge cases needing LLM judgment get rigid "skip in this case" rules (Form A). Both are wrong. 🛡 How to Avoid - Classify all harness operations as "requires judgment" vs. "can execute deterministically" and make the boundary explicit - Enforce test execution, lint, build, and gate decisions in deterministic code — don't "ask nicely" - Restrict LLM use to genuinely judgment-dependent parts (root cause analysis, strategy decisions, code generation) - Periodically review and adjust the boundary as models evolve #HarnessEngineering# #AIAgent#

Harness Engineering Practices P20. Make Legibility and Calibrated Uncertainty an SLO 🎯 Point Now that generation is cheap, the real bottleneck is "human review time." Diffs should be optimized not just for correctness, but for reviewability. 📝 Overview Optimize diffs for review time, not just correctness. Small, focused PRs. Explanations that tell "why." Explicit flagging of risky areas. Additionally, have the agent explicitly output "areas of low confidence" so the harness can route them to additional verification or human review. Calibrated uncertainty is more valuable than false confidence. 🔍 Explanation As agent generation speed increases, the bottleneck shifts from "writing code" to "reviewing code." Giant PRs, unexplained changes, confidently-presented but actually uncertain implementations — these explosively consume reviewer time. Treating legibility as an SLO (Service Level Objective) and measuring/optimizing PR size, explanation presence, and change rationale improves overall throughput. Having agents explicitly state "I'm not confident here" and "this needs human verification" lets reviewers focus on what matters. This is the most overlooked practice for building trustworthy autonomous agents. 🛠 How to Practice - Add "change rationale," "confidence level (high/medium/low)," and "review focus areas" fields to PR templates and require the agent to fill them - Set PR size limits and force splitting when exceeded - Require "low confidence" markers in agent output so the harness can route those areas to additional verification - Measure review time per PR and identify causes of long reviews (giant diffs, missing explanations, etc.) for improvement 💼 Use Cases - Issue-to-PR agents including change rationale and confidence markers in PRs - Code review agents suppressing low-confidence trivial comments and focusing on types humans miss - Migration: keeping per-unit PRs small and focused to distribute reviewer load ⚠ Pitfalls Over-optimizing for legibility can make agent output overly conservative. "Uncertainty expression" can also become noise — an agent that says "I'm not confident" about everything is useless. Calibration is key: accurately marking only genuinely uncertain areas is what creates value. Don't forget to measure review time either. Quantitatively tracking whether PR spam or giant diffs are crushing review bandwidth is the starting point for improvement. #HarnessEngineering# #CodeReview#

Harness Engineering Practices P19. Turn Failures into Data Assets (The Harness's Own Retro Loop) 🎯 Point A harness that repeats the same failures is an unimprovable black box. Log failures and feed them back into harness improvement. 📝 Overview Log every human intervention, rollback, and escaped defect with its cause, then feed this into harness improvements (new gates, new instructions, new tools). The harness should have CI for itself. This is what separates maturity L3 (measuring) from L4 (continuously improving). 🔍 Explanation When agents fail, most organizations conclude "the model is bad." But the real question is "why couldn't the harness prevent this failure?" A human intervention means the harness lacked a guardrail or verification. A needed rollback means the circuit breaker didn't trigger. An escaped defect means the verifier was insufficient. Recording these events with root cause analysis and converting them into harness improvement actions (adding gates, updating instruction files, improving tools) is the loop that matures a harness into a product. 🛠 How to Practice - Record all human interventions, rollbacks, and escaped defects in structured logs with cause classification - Run regular retrospectives (weekly or biweekly) to identify recurring failure patterns - For each failure pattern, select and implement the most effective improvement action (new gate, instruction addition, tool improvement) - Measure improvement action effectiveness and retract low-impact ones to try different approaches 💼 Use Cases - Weekly analysis of issue-to-PR agent failures to identify harness improvement points - CI auto-maintenance: tracking false positive causes to improve triage logic - Incident response: deriving observability access improvements from cases where agent recommendations were inaccurate ⚠ Pitfalls Adding a rule after every failure leads to "Scaffolding Ratchet" (AP3). Turning failures into data assets isn't about adding more rules — it's about root cause analysis and choosing the most effective improvement. Logging without analysis accumulates data without generating value. Regular retrospective processes are essential. #HarnessEngineering# #ContinuousImprovement#

Harness Engineering Anti-Patterns AP10. The Unobservable Black Box 🎯 Point Was it the model, the prompt, the tool, the context, or the environment? Nobody can tell. A harness where improvement runs on superstition and intuition is a harness that cannot improve. ❗ Problem Failures can't be attributed to subsystems, so no one knows what to fix. Improvement becomes superstition and guesswork, and the harness's retrospective loop stops turning. Combined with "metric monoculture" — tracking only a single metric — unmeasured qualities silently degrade. 🔍 Mechanism & Symptoms Observability is unglamorous infrastructure work, and agents "mostly work," so this anti-pattern gets deprioritized. But without the ability to attribute failures to subsystems, the retrospective loop can't run and the harness becomes unimprovable. Further, tracking a single metric (e.g., success rate only) creates "metric monoculture" where unmeasured virtues (review time, regression rate, code maintainability) quietly suffer. Symptoms: "why did it fail? no idea" is frequent, improvement efforts default to "tweak the prompt," model vs. harness issues are indistinguishable, and success rate improves while reviewer frustration grows. 📋 Scenarios - An agent fails a task but no one can determine whether it was model reasoning error, insufficient context, a tool bug, or an environment issue. The team repeats "let's make the prompt more detailed" as symptomatic treatment. - Success rate is tracked as the sole metric. It improves to 80%, but no one notices that review time for successful cases has tripled. - A model upgrade shows no performance change. Whether it's a model issue or harness scaffolding constraining performance (AP3) can't be distinguished, and investment decisions become superstition. 🛡 How to Avoid - Trace decisions, tool calls, and context transitions, making them attributable to the 7 subsystems (perception, action, feedback, control, memory, guardrails, interface) - Measure in bundles, not single metrics (success rate, intervention rate, rework rate, regression rate, cost, review time, confidence calibration) - A/B test harness changes with the model held fixed to attribute improvements to the harness - Invest in harness observability as "unglamorous but essential infrastructure" and build it as the foundation for improvement loops #HarnessEngineering# #AIAgent#

Harness Engineering Practices P17. Dry Run by Default and Blast Radius Preview 🎯 Point "Restarting 40 pods" should be presented before execution, not discovered after. Preview side effects before they happen. 📝 Overview Actions with side effects first show a preview of their effects (which files, which lines, which pods), then pass through a gate or human confirmation before execution. Default to dry run (show results without executing), and execute only after explicit approval. 🔍 Explanation Knowing "what the agent will do" in advance is foundational to safety. Diff previews, lists of affected services, pod restart counts, message contents — presenting these before execution enables informed human judgment. In incident response especially, presenting remediation blast radius and rollback plans together, then executing with monitoring post-approval, is an effective pattern. Making dry run the default structurally eliminates "accidental execution" risks. 🛠 How to Practice - Implement "preview mode" for all side-effecting operations, displaying impact scope before execution - Include quantitative info in dry run output: number of affected files, pods, changed lines, etc. - Make dry run mandatory for irreversible operations; keep it optional for reversible ones to maintain efficiency - Record dry run vs. actual execution discrepancies and continuously improve dry run accuracy 💼 Use Cases - Incident response: present blast radius and rollback plan for remediation actions upfront - Pair programming: display agent edits as diff previews before applying - Migration: canary-apply changes to a subset first and verify before full rollout ⚠ Pitfalls Dry runs aren't always accurate — environmental differences can cause failures at execution time that didn't appear in dry run. If dry runs become reflexively skipped, they lose meaning. Combine with P15 (revocability): execute reversible operations without dry run, require dry run only for irreversible operations. #HarnessEngineering# #AIAgent#

Harness Engineering Practices P15. Design Permissions Around Revocability, Not Capability 🎯 Point The question isn't "can it do X" but "can X be undone?" Revocability is the right axis for cutting the autonomy-safety tradeoff. 📝 Overview Freely permit reversible actions (commits to branches, file edits — git can undo these). Gate only irreversible actions (force-push, production writes, external emails, payments). Revocability is the criterion for permission design. 🔍 Explanation Most permission designs rely on "does this feel dangerous" intuition, which is often wrong. File editing looks "dangerous" but is instantly reversible via git, making actual risk low. An external API request looks "trivial" but can't be unsent, making actual risk high. Using revocability as the criterion naturally yields optimal autonomy-safety balance. Requiring approval for reversible operations causes "Gate Fatigue" (AP8) — humans rubber-stamp reflexively, missing truly dangerous operations. 🛠 How to Practice - List all operations the agent can perform and classify each as "reversible," "irreversible," or "conditionally reversible" - Permit reversible operations (branch commits, file edits, sandbox execution, etc.) without approval - Require approval gates for irreversible operations (force-push, production writes, external API calls, email sends, etc.) - Consider effect chains (a commit that triggers CI that triggers auto-deploy creates indirect irreversibility) 💼 Use Cases - Issue-to-PR agents: commits to branches are free, force-push is forbidden - Incident response: diagnosis (reads) is autonomous, remediation (production writes) requires human approval - Prototyping: sandbox operations are free, only external network access is gated ⚠ Pitfalls Judging "reversible" isn't always simple. A branch commit is reversible, but if that branch auto-triggers CI which triggers deployment, it has indirectly irreversible effects. Consider the full chain of effects when judging revocability. Don't over-rely on "it can be undone" either — the cost of undoing matters too. #HarnessEngineering# #AIAgent#

Harness Engineering Anti-Patterns AP1. The Context Hoarder 🎯 Point "Include everything just in case" — that feeling of safety is silently killing your agent's performance. More information isn't safer; it's often harmful. ❗ Problem The context window overflows with irrelevant information, diluting the agent's attention. Important information gets buried in the middle, leaving insufficient space for the code and specs the agent actually needs. The result: degraded judgment accuracy, with cost and latency worsening super-linearly. 🔍 Mechanism & Symptoms This anti-pattern is seductive because the intuition "more info = safer" is strong. Designing retrieval (what to fetch and when) takes effort, so stuffing everything in feels easier. But the context window is a scarce resource like a CPU's L1 cache. Utility does not increase monotonically. Beyond a threshold, irrelevant tokens dilute the attention mechanism, causing "lost in the middle" — critical info buried in the center gets ignored. Symptoms include injecting entire repositories, full conversation histories, all tool definitions on every call, and "the agent inexplicably ignores information it already read." 📋 Scenarios - An issue-to-PR agent is fed not just the issue but the entire repo's README, config files, and past PR history. The agent misses the issue's key points and starts editing unrelated files. - A migration agent receives thousands of files at once, overflowing context and losing consistency mid-task. - In pair programming, unopened files and long conversation history consume context, slowing responses. 🛡 How to Avoid - Measure context usage by category and visualize allocation like a memory profiler - Default to pull (let the agent fetch what it needs) and limit push (force injection) to only invariants that are fatal to violate - Summarize and compress older conversation turns; dynamically load only task-relevant tool definitions - When you catch yourself thinking "include everything for safety," recognize that as this anti-pattern's signature #HarnessEngineering# #AIAgent#

Harness Engineering Practices P12. Semantic Circuit Breakers and "Discard Context" Restarts 🎯 Point Ever seen an agent repeat the same error three times? "Push through" is far slower than "restart with a lesson learned." 📝 Overview Instead of simple max-iteration limits, semantically detect "same error twice," "oscillating between two edits," or "no net progress" and halt. When stuck, don't keep kneading in contaminated context. Roll back to the last good checkpoint, discard failure history from context, carry just a one-line lesson, and restart with a fresh approach. 🔍 Explanation When an agent gets stuck, the least efficient response is "try harder in the same context." As failure history accumulates in context, the agent gets dragged toward repeating the same mistakes. Semantic circuit breakers judge "is there actual progress" rather than counting iterations. When they detect oscillation (change A to B then back to A) or repeated identical errors, they trigger a git rollback to the last clean state, then restart in fresh context with only "last time X failed, try a different approach." Git is the agent's Undo. 🛠 How to Practice - Implement logic in the harness that compares error messages and test results to detect "same error twice" and "A-B edit oscillation" - On halt, git checkout to the last clean commit and discard contaminated context - On restart, carry only a one-line lesson ("last time X failed") into the fresh context - Tune circuit breaker thresholds per task difficulty (strict for easy tasks, more lenient for hard ones) 💼 Use Cases - Issue-to-PR agents looping on the same test failure they can't fix - Compile error fixing: detecting fix-then-break-then-revert cycles - Legacy code modernization: recognizing when a fundamentally different approach is needed ⚠ Pitfalls Too-sensitive circuit breakers cut off agents one step from a solution. Too-lenient ones waste cost in infinite loops. Tuning the definition of "progress" per task is critical. Distilling accurate lessons at restart time is also hard — wrong lessons carried into a restart can trigger different failure patterns. #HarnessEngineering# #AIAgent#