Harness Engineering Anti-Patterns AP3. The Scaffolding Ratchet 🎯 Point A rule is added after every failure, but nothing is ever removed. Before you know it, your Rube Goldberg harness is fighting the model. "It worked once" doesn't mean "it's still needed." ❗ Problem Rules, steps, and guardrails accumulate endlessly, turning the harness into a complex labyrinth. Even as model capabilities improve, legacy scaffolding creates performance ceilings. No one grasps the full picture, making improvement nearly impossible. 🔍 Mechanism & Symptoms This anti-pattern spreads because each rule addition is locally justified ("it prevented that one incident") and deletion feels risky. But scaffolding becomes a one-way ratchet, degrading into a Rube Goldberg machine. As models get smarter, legacy scaffolding unnecessarily constrains their reasoning, becoming a performance ceiling. Symptoms include: no one knows all the rules, new rules contradict old ones, model upgrades don't improve performance (scaffolding bottlenecks), and the harness becomes untouchable because modifications feel too risky. 📋 Scenarios - One bug leads to "always read file A first." Another adds "read file B first." A third adds "always split plans into 3 phases." The agent now follows unnecessary procedures on every task, slowing down even simple ones. - Upgrading from GPT-4 to Claude Opus shows no improvement because scaffolding added for GPT-4's weaknesses suppresses Claude Opus's strengths. - The harness rule file bloats to 500 lines, incomprehensible to new team members. Improvement proposals are rejected with "there was an incident before." 🛡 How to Avoid - Introduce garbage collection for scaffolding — audit all rules on every model update and remove unnecessary ones - Record "why added," "when added," and "which model version" for each rule - Periodically benchmark with scaffolding removed to confirm which pieces are truly necessary - Consciously practice "rule deletion" as an improvement action, not just "rule addition" #HarnessEngineering# #AIAgent#