If agents leave GPUs idle while waiting on tools, a scheduler should look at when those tools return. TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving 🔎 Overview A scheduler for agents that alternate between LLM inference and tool calls. It scores tool-return timing together with KV-cache reuse, then picks which sessions to admit via combinatorial optimization. ⚠️ The problem ・Existing schedulers don't use tool-return timing directly ・Greedy per-session decisions can block a better combination of sessions competing for the same capacity 🛠 Method ・Each session gets three values compared in order: time-decayed cache credit (10 s half-life), tool-return readiness, and token count ・A capacity-state dynamic program jointly selects waiting sessions and assigns nodes ・The DP runs in an async worker and is revalidated against live state before applying; overhead is under 0.12% of runtime 📊 Results ・On DeepSeek-V4 Flash, SWE-Lite, 64 workers, vs. the best baseline: up to 26.8% lower latency, 36.3% lower TTFT, 29.5% higher decode throughput ・Swapping the DP for BFD drops decode from 1417.97 to 1233.72 tok/s ・It doesn't win everywhere: TTFT gets worse on AutomationBench at 128 workers #LLMServing# #AIAgents#