Harness Engineering Practices P17. Dry Run by Default and Blast Radius Preview 🎯 Point "Restarting 40 pods" should be presented before execution, not discovered after. Preview side effects before they happen. 📝 Overview Actions with side effects first show a preview of their effects (which files, which lines, which pods), then pass through a gate or human confirmation before execution. Default to dry run (show results without executing), and execute only after explicit approval. 🔍 Explanation Knowing "what the agent will do" in advance is foundational to safety. Diff previews, lists of affected services, pod restart counts, message contents — presenting these before execution enables informed human judgment. In incident response especially, presenting remediation blast radius and rollback plans together, then executing with monitoring post-approval, is an effective pattern. Making dry run the default structurally eliminates "accidental execution" risks. 🛠 How to Practice - Implement "preview mode" for all side-effecting operations, displaying impact scope before execution - Include quantitative info in dry run output: number of affected files, pods, changed lines, etc. - Make dry run mandatory for irreversible operations; keep it optional for reversible ones to maintain efficiency - Record dry run vs. actual execution discrepancies and continuously improve dry run accuracy 💼 Use Cases - Incident response: present blast radius and rollback plan for remediation actions upfront - Pair programming: display agent edits as diff previews before applying - Migration: canary-apply changes to a subset first and verify before full rollout ⚠ Pitfalls Dry runs aren't always accurate — environmental differences can cause failures at execution time that didn't appear in dry run. If dry runs become reflexively skipped, they lose meaning. Combine with P15 (revocability): execute reversible operations without dry run, require dry run only for irreversible operations. #HarnessEngineering# #AIAgent#

Harness Engineering Practices P16. Trust Through Earned Credentials (Capability Ramping) 🎯 Point No company gives a new hire production deploy access on day one. Agents need a system for "earning trust through track record" too. 📝 Overview New harnesses start with strict gates. When an agent builds a track record on a task type (high success rate, zero regressions), autonomy expands for that type only. Trust is earned per task type based on measurement, and is always revocable. 🔍 Explanation Granting full autonomy for all tasks from the start is dangerous, but maintaining strict gates forever is inefficient. The solution is gradual trust earning. For example, after 10 consecutive successes with zero regressions on lint fixes, permit auto-merge for lint fixes only. Security patches stay gated until trust is earned on a separate track. The key principles: trust is independent per task type, based on measurement, and always retractable. When a regression occurs, immediately downgrade trust for that task type. 🛠 How to Practice - Define trust levels per task type (e.g., L0 = all approvals required, L1 = light autonomy, L2 = auto-merge allowed) - Define promotion criteria numerically (e.g., 95%+ success rate over last 20 tasks with zero regressions) - Define demotion criteria too (immediately drop to L0 on regression) - Log trust level changes and their rationale to ensure decision transparency 💼 Use Cases - CI auto-maintenance: gradually permit auto-merge for low-risk types (lint patches) only - Issue-to-PR agents: adjust human review granularity based on bug fix success track record - Legacy code modernization: characterization test creation is autonomous; code changes proceed cautiously with canary deploys ⚠ Pitfalls Unclear trust-earning criteria lead to either permanent strict gates or unnoticed drift toward laxity. Define criteria numerically (success rate above X%, zero regressions in last N tasks) with automated evaluation. Also, "high success rate" doesn't equal "safe" — consider whether the agent is padding stats with easy tasks. #HarnessEngineering# #AIAgent#

Harness Engineering Practices P14. Independent Self-Critique Pass (Separate Generator and Critic Context) 🎯 Point Do you trust an agent that reviews its own code and says "looks good"? Reviews only function when separated from the author. 📝 Overview After implementation, critique the diff against the spec in a separate context (ideally with a different role prompt). To avoid the "I just wrote it so it must be correct" bias, the generator and critic must not share context. 🔍 Explanation For the same reason human code reviews are performed by someone other than the author, agent self-critique requires context separation. When generation and critique happen in the same context, the "why I wrote it this way" bias carries directly into critique, making problems easier to miss. In a separate context — ideally a different prompt or model instance — showing only the diff and spec enables a fresh-perspective review. This adds computational cost, but the defect-prevention payoff is significant. 🛠 How to Practice - After implementation, launch a separate context (new conversation session) and provide only the diff and spec for review - Give critics focused roles ("spec divergence," "security risks," "regression potential," etc.) - Where possible, use different models or prompts for generation and critique to ensure perspective diversity - Have critique results output in structured format so the harness can auto-decide actions (send fix instructions, report to humans) 💼 Use Cases - Issue-to-PR agents running a self-review with a different prompt before completion - Autonomous bug fixing: validating fix soundness in a separate context - Code generation: running security-focused reviews as a separate agent ⚠ Pitfalls A critic designed to "flag everything" produces a flood of trivial comments, creating infinite implement-fix-critique-fix loops. Give critics focused roles: "spec divergence," "security risks," "regression potential." Low-quality critics also risk unnecessarily modifying correct implementations. #HarnessEngineering# #CodeReview#

Harness the Sun. Reach Kardashev Level II. Terrestrial and orbital AI data centers at a scale orders of magnitude beyond today. If anyone on Earth can pull it off, it’s Elon, C.C. Wei, and Jensen. Terafab × TSMC × Nvidia.

Harness Engineering Practices P12. Semantic Circuit Breakers and "Discard Context" Restarts 🎯 Point Ever seen an agent repeat the same error three times? "Push through" is far slower than "restart with a lesson learned." 📝 Overview Instead of simple max-iteration limits, semantically detect "same error twice," "oscillating between two edits," or "no net progress" and halt. When stuck, don't keep kneading in contaminated context. Roll back to the last good checkpoint, discard failure history from context, carry just a one-line lesson, and restart with a fresh approach. 🔍 Explanation When an agent gets stuck, the least efficient response is "try harder in the same context." As failure history accumulates in context, the agent gets dragged toward repeating the same mistakes. Semantic circuit breakers judge "is there actual progress" rather than counting iterations. When they detect oscillation (change A to B then back to A) or repeated identical errors, they trigger a git rollback to the last clean state, then restart in fresh context with only "last time X failed, try a different approach." Git is the agent's Undo. 🛠 How to Practice - Implement logic in the harness that compares error messages and test results to detect "same error twice" and "A-B edit oscillation" - On halt, git checkout to the last clean commit and discard contaminated context - On restart, carry only a one-line lesson ("last time X failed") into the fresh context - Tune circuit breaker thresholds per task difficulty (strict for easy tasks, more lenient for hard ones) 💼 Use Cases - Issue-to-PR agents looping on the same test failure they can't fix - Compile error fixing: detecting fix-then-break-then-revert cycles - Legacy code modernization: recognizing when a fundamentally different approach is needed ⚠ Pitfalls Too-sensitive circuit breakers cut off agents one step from a solution. Too-lenient ones waste cost in infinite loops. Tuning the definition of "progress" per task is critical. Distilling accurate lessons at restart time is also hard — wrong lessons carried into a restart can trigger different failure patterns. #HarnessEngineering# #AIAgent#

Harness Engineering Practices P10. Two-Phase Design — Read-Only Exploration, Then Implementation 🎯 Point "Understand before you write" is a human virtue. Enforce the same discipline on agents — not with wishes, but with permissions. 📝 Overview Force a write-locked exploration phase first, requiring a plan before entering the implementation phase. Structurally prevent premature editing to improve plan quality. Phase boundaries are enforced by the harness through permissions, not by prompting. 🔍 Explanation Agents tend to "just start writing." They edit files before grasping the full picture, then realize "the approach was wrong to begin with," causing massive rework. Two-phase design permits only file reads, searches, and symbol resolution in the first phase, disabling edit tools. The agent focuses solely on exploring, understanding, and planning. Edit permissions unlock only after the plan is approved. This structurally eliminates the risk of "impulsive edits that break things." 🛠 How to Practice - In Phase 1, disable edit tools and configure a tool set permitting only file reads, searches, and symbol resolution - Require a plan file (plan.md) as Phase 1 output and make plan approval the gate for entering Phase 2 - Enforce phase boundaries at the tool permission level, not through prompt-based requests - Set time and step limits on the exploration phase to prevent context exhaustion 💼 Use Cases - Issue-to-PR agents: analyze the issue and draft a plan in read-only mode, implement only after plan approval - Legacy code modernization: map and understand modules first, then apply changes - Incident response: separate diagnosis (read-only, safe, autonomous) from remediation (write, gated) ⚠ Pitfalls An overly long exploration phase can exhaust the context window. Knowledge gathered during exploration may also become stale by implementation time (combine with P3's TTL). "Asking nicely in the prompt" isn't sufficient for phase boundaries — enforce them at the tool permission level. #HarnessEngineering# #AIAgent#

Harness optimization is getting real receipts. AutoSaddler is offline harness learning from agent failure traces. Not another prompt tweak loop. It patches prompts, tools, and middleware as code. Then it keeps updates that survive a held out set. On the test sets (Pass@1): GAIA2: 53.0 → 62.0 (+9.0) SWE-Bench Pro: 37.3 → 46.9 (+9.6) Terminal-Bench 2.0: 40.0 → 50.0 (+10.0) That TB2 number also clears the expert tuned Terminus KIRA at 47.5. Kill generalization aware selection and GAIA2 falls to 50.6, under the default agent. Deep diagnosis and structured patches help. Dev set filtering is what stops the harness from overfitting the mini batch. On GAIA2, Figure 1b, about 147 leveraged traces to the best dev score vs about 1,400 for Meta-Harness.

Harness Engineering Anti-Patterns AP8. Gate Fatigue / Rubber-Stamping 🎯 Point Approval dialogs even for reversible operations, dozens of reflexive "approve" clicks per day. Excessive gates neutralize the gates themselves. ❗ Problem Too many approval requests cause humans to reflexively skip gates. Through the same mechanism as alert fatigue, even truly dangerous irreversible action approvals get rubber-stamped, completely defeating the purpose of gates. 🔍 Mechanism & Symptoms "Human approval on everything" looks safe and reassures auditors, making this anti-pattern organizationally popular. But meaningless approval repetition makes approval reflexive — identical to alert fatigue. If you're asked "may I edit this file?" 50 times a day, the 51st "may I delete the production DB?" gets the same reflexive approval. Symptoms: approval time under 1 second (proof of not reading), all approvals logged as "reviewed" but actually unreviewed, important operation approvals buried among trivial ones, and team complaints about "annoying approvals." 📋 Scenarios - The agent requests approval for every file read. Developers habitually select "allow all," and later reflexively approve when the agent executes a force-push. - Security team mandates "approval required for all operations." Developers process 200 approvals daily and create a script to batch-approve everything. Gates are completely hollow. - In pair programming, every agent edit requires diff preview and approval. Developers mash Enter without reading diffs, and unintended changes slip through. 🛡 How to Avoid - Design permissions around "revocability" not "capability" — freely permit reversible operations - Consolidate only irreversible operations into "heavyweight ceremony" gates, preserving gate scarcity - Measure approval response times — sub-1-second consistently indicates gate decay - Consciously minimize gate count, limiting them to "moments that truly require human judgment" #HarnessEngineering# #AIAgent#