# Practices for Embedding AI Agents in Software # Adaptive Timeout & Budget-Bounded Retry 🎯 The Hook Still retrying exactly 3 times? A single LLM retry can burn thousands of tokens. In the agent era, retry limits should be budgets, not counters. 🔥 The Problem Agent execution mixes operations with wildly different latency profiles: tool calls (seconds) vs. LLM inference (tens of seconds to minutes). A single fixed timeout either waits too long for tools or cuts off inference too early. Fixed-count retries ignore cost: three LLM retries can blow through your token budget, while three network retries may not be enough. And treating a 429 rate-limit the same as a schema violation wastes resources on retries that cannot succeed. 💡 The Pattern Adaptive Timeout & Budget-Bounded Retry sets timeouts per operation class (tool call, LLM inference, full session) and caps retries by remaining token budget rather than a fixed count. Errors are classified into three types: transient (429, 5xx) handled with exponential backoff, content-caused (schema violations) handled with self-correction, and context overflow handled with summarization or splitting. As budget consumption crosses thresholds, the system degrades gracefully: falling back to lighter models, returning partial results, and ultimately failing fast. ✅ When to Use Use when: - The agent combines operations with different latency characteristics - Retry cost is non-trivial (includes LLM inference) - Both transient network errors and content-caused errors can occur Don't use when: - A single LLM call with a fixed timeout is sufficient - Retries are prohibited (non-idempotent writes without idempotency keys) ⚠️ Pitfalls - Ignoring the Retry-After header on 429 responses and relying solely on your own backoff will trigger further throttling by the provider - Retrying non-idempotent writes without idempotency keys causes double execution. Either add keys or skip retries entirely for those operations - Stuffing raw error messages into self-correction prompts bloats the context window and triggers context-length errors. Summarize error feedback to roughly 200 tokens 🔧 Implementation Approach - Define timeouts per operation class: roughly 10-30s for tool calls, 60-120s total for LLM inference (with 5-15s inter-token timeout during streaming), and session-level deadlines for the full execution - Classify errors into three types and route accordingly: transient errors (429/5xx/timeout) use exponential backoff with jitter, content errors (schema violations etc.) trigger self-correction by appending an error summary to context, and context overflow triggers summarization or splitting - Cap retries by remaining token budget rather than a fixed count. Allow transient retries up to 90% of the budget and self-correction retries up to 70% - Deploy independent circuit breakers (Closed/Open/Half-Open) per provider, and on breaker open, descend a degradation ladder: lighter model, cached response, static fallback, then fail-fast - Implement a provider abstraction layer with a common interface so fallback routing is transparent to callers, and attach a degradation-level tag to response metadata #AIAgents# #SoftwareArchitecture#