Harness Engineering Practices
P16. Trust Through Earned Credentials (Capability Ramping)
🎯 Point
No company gives a new hire production deploy access on day one. Agents need a system for "earning trust through track record" too.
📝 Overview
New harnesses start with strict gates. When an agent builds a track record on a task type (high success rate, zero regressions), autonomy expands for that type only. Trust is earned per task type based on measurement, and is always revocable.
🔍 Explanation
Granting full autonomy for all tasks from the start is dangerous, but maintaining strict gates forever is inefficient. The solution is gradual trust earning. For example, after 10 consecutive successes with zero regressions on lint fixes, permit auto-merge for lint fixes only. Security patches stay gated until trust is earned on a separate track. The key principles: trust is independent per task type, based on measurement, and always retractable. When a regression occurs, immediately downgrade trust for that task type.
🛠 How to Practice
- Define trust levels per task type (e.g., L0 = all approvals required, L1 = light autonomy, L2 = auto-merge allowed)
- Define promotion criteria numerically (e.g., 95%+ success rate over last 20 tasks with zero regressions)
- Define demotion criteria too (immediately drop to L0 on regression)
- Log trust level changes and their rationale to ensure decision transparency
💼 Use Cases
- CI auto-maintenance: gradually permit auto-merge for low-risk types (lint patches) only
- Issue-to-PR agents: adjust human review granularity based on bug fix success track record
- Legacy code modernization: characterization test creation is autonomous; code changes proceed cautiously with canary deploys
⚠ Pitfalls
Unclear trust-earning criteria lead to either permanent strict gates or unnoticed drift toward laxity. Define criteria numerically (success rate above X%, zero regressions in last N tasks) with automated evaluation. Also, "high success rate" doesn't equal "safe" — consider whether the agent is padding stats with easy tasks.
#
HarnessEngineering# #
AIAgent#