Round-robin load balancing built for chat can't keep up with agent inference anymore. Title: Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs URL: 📌 Overview llm-d, an open-source distributed inference stack, hits 1.0. It turns vLLM and SGLang servers on Kubernetes into one serving pool. It is a CNCF Sandbox project with 500+ contributors. ❓ The problem Agents share long prefixes and arrive in bursts, causing redundant computation, memory pressure, and interference with interactive users. 🛠️ Key mechanisms ・Cache-aware routing with priority-based flow control ・KV cache offload to host memory, SSD, and Managed Lustre ・Disaggregated prefill/decode with safe rolling updates via DisaggregatedSet ・Time-slicing that fills idle RL capacity with async inference 📊 Results (vendor-reported) ・Agentic coding on TPU: median TTFT down 85%+, throughput 2.4x ・Cache hit rate from 35% to 70% ・About 60% fewer GPU-hours on A3 Mega ・RL duty cycle from ~40% to 70%+ 🌍 Use cases Snap, ZenteiQ, and Mistral use it. You can start by putting prefix-aware routing in front of an existing vLLM deployment. #llmd# #LLMInference#