# Practices for Embedding AI Agents in Software # Model Router & Adaptive Effort 🎯 The Hook Running every request through your most powerful model is like dispatching a senior engineer to answer every support ticket. Route by difficulty, and you cut costs dramatically without sacrificing quality. 🔥 The Problem Processing all requests with the highest-performance model quickly blows through monthly budgets. But routing everything to a lightweight model causes quality collapse on complex reasoning and planning tasks. Large models also carry higher latency and heavier tail distributions, so using them for simple tasks unnecessarily degrades system-wide response times. Without model tiering, you're stuck in a cost-vs-quality tradeoff with no middle ground. 💡 The Pattern Dynamically select the model based on task difficulty, type, and risk. Start with a lightweight model; if confidence is low, escalate to a higher-tier model. Begin with a rule-based router (input length, task type, keywords) and upgrade to a classifier only if accuracy is insufficient. A 2-3 tier setup (small, medium, large) is a practical starting point. In production, 60-80% of requests typically resolve at the small or medium tier. ✅ When to Use Use when: - Task types are diverse, mixing routine extraction/classification with complex reasoning - A clear monthly cost ceiling exists - Traffic volume is high enough to recoup the routing mechanism's cost Don't use when: - All requests are similar difficulty -- a single model suffices - Quality can't drop even 1% on any request -- always use the top model plus caching - Request volume is low (hundreds per month) and routing development cost exceeds savings ⚠️ Pitfalls - Don't ignore the router's own cost. An LLM-based router can cost as much as one lightweight model call. Start with rule-based routing - Define confidence precisely. Use measurable indicators -- structured output parse error rate, refusal rate, internal log probabilities. "Seems uncertain" isn't operational - Prevent escalation infinite loops. Set a termination condition when even the top-tier model returns low confidence -- fall back to human escalation or error response 🔧 Implementation Approach - Start with a rule-based router (input length, task type, keywords) and only upgrade to a meta-classifier when classification accuracy proves insufficient - Abstract model tiers (small, medium, large) behind a common interface so providers can be swapped without changing routing logic - Run a confidence check on the lightweight model's response (parse error rate, refusal rate, log probabilities) and auto-escalate to a higher tier when confidence falls below the threshold - Cap escalation at the top tier -- if even the largest model returns low confidence, fall back to human escalation or an error response - Monitor routing ratios, cost, and latency on an ongoing basis to detect drift from model version changes and re-calibrate thresholds accordingly #AIAgents# #SoftwareArchitecture#