# Decision Points in AI Agent Development # Timeout 🎯 The Hook Still using a flat 30-second timeout for everything? In AI agent systems, tool calls take seconds, LLM inference takes tens of seconds, and full sessions can run for tens of minutes. A one-size-fits-all timeout simply doesn't work. Layered timeouts are the starting point. 📋 Overview Timeout controls the maximum time an agent waits for a specific operation to complete. Traditional web services could get away with a single HTTP timeout of ~30 seconds, but AI agents are different: individual requests are long-running, latency variance is high, and provider availability is unstable. Too short, and you kill legitimate processing during temporary slowdowns. Too long, and you mask failures while tying up resources. The goal is to set timeouts per operation class and dynamically adjust them based on observed data. 🔍 Decision Points Timeout values primarily reflect the latency_budget -- how long users or downstream systems can wait. Set timeouts across at least three layers 📐 1. Tool call layer: External APIs, database queries, file operations. Further subdivide by tool type for precision 2. LLM inference layer: Varies heavily with input token count and model load. Small vs. large models can differ by 10x in response time 3. Session-level layer: Work backward from "how many minutes will the user wait for this task?" Observe P99 latency for each operation and apply a safety multiplier of 1.5-2.0x as a starting point. In early stages without P99 data, use reference values and switch to empirical data after 1-2 weeks of production traffic. 💡 Key Details Reference values 📊 - Tool calls (general): 10-30s. Database queries may need as little as 5s - LLM inference (overall): 60-120s. Increase ceiling for long-form generation tasks - LLM inference (inter-token): 5-15s. Critical for early detection of provider hangs during streaming - LLM inference (TTFT): 15-30s. Scales with input length - Full session: Minutes to tens of minutes. Shorter for interactive, longer for research/analysis When using streaming, inter-token timeout is more essential than overall timeout as a monitoring signal 🔍 Normal inference returns tokens every few hundred milliseconds to a few seconds, so 15 seconds of silence strongly suggests a provider-side anomaly. ⚖️ Trade-offs Too short, and you interrupt complex reasoning mid-generation ⏱️ Deep Chain of Thought processes, large context prefill, and peak-hour provider delays all trigger premature timeouts. This spawns retry cascades that compound the problem. Too long, and unresponsive providers hold threads for minutes, blocking other requests 🚫 Hung tool calls go undetected, users endure the worst UX -- waiting with no end in sight. Failure detection is delayed, preventing circuit breakers from activating when they should. Timeout changes directly impact retry strategy. Shorter timeouts increase timeout frequency, which increases retries. Always tune timeouts in concert with retry budgets and session-level budgets. 🛠️ Use Cases Interactive chat assistant 💬 Users expect responses within seconds. Tools at 3-5s, LLM inference at 15-30s (streaming tokens immediately), session at 60-120s. Streaming drastically reduces perceived latency. Research and analysis agent 🔬 Minutes of processing are acceptable. Tools at 10-60s, LLM inference at 60-180s, session at 5-30 minutes. Cost and step budgets become the binding constraint, not timeouts. Automated code review agent 💻 Large diffs push input tokens into the tens of thousands, extending TTFT significantly. Dynamically adjusting TTFT timeout based on input token count is highly effective here. Timeout is not cancellation ⚠️ A timeout means the client stops waiting, but the provider may continue processing. Since billing is based on provider-side compute, send a cancellation request as well when the API supports it. #AIAgents# #SoftwareArchitecture#