Harness Engineering Anti-Patterns
AP7. The Misplaced Determinism Boundary
🎯 Point
Trusting the LLM's goodwill to run tests, while cramming edge case judgment into rigid rules. Get the boundary wrong and you lose both reliability and adaptability.
❗ Problem
Putting probabilistic elements where determinism is needed leaks reliability; putting deterministic rules where judgment is needed loses adaptability. The result: instability on simple tasks, rigidity on complex ones, or both simultaneously.
🔍 Mechanism & Symptoms
This misplacement takes two forms. Form A: cramming LLM-judgment long tails into rigid rules. Rules feel predictable, but break brittly on edge cases. Form B: entrusting deterministic operations (test execution, gate decisions, retries) to LLM goodwill. Delegating to the model feels easier, but produces instability — tests forgotten 1 in 100 runs. Symptoms of Form A: "rules break on unexpected cases," "need new rules for every new pattern." Form B: "test execution forgotten," "gates skipped," "randomly stops working."
📋 Scenarios
- Form A: A rigid rule "import changes must be at file top" is set. When circular import resolution requires otherwise, the rule blocks the agent and it gets stuck.
- Form B: "Always run tests" is stated in the prompt but not harness-enforced. The agent runs tests 95% of the time but declares completion without tests the other 5%.
- Both: Most codemod work could be deterministic AST transforms, but everything is delegated to the LLM (Form B). Meanwhile, edge cases needing LLM judgment get rigid "skip in this case" rules (Form A). Both are wrong.
🛡 How to Avoid
- Classify all harness operations as "requires judgment" vs. "can execute deterministically" and make the boundary explicit
- Enforce test execution, lint, build, and gate decisions in deterministic code — don't "ask nicely"
- Restrict LLM use to genuinely judgment-dependent parts (root cause analysis, strategy decisions, code generation)
- Periodically review and adjust the boundary as models evolve
#
HarnessEngineering# #
AIAgent#