Round-robin load balancing built for chat can't keep up with agent inference anymore. Title: Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs URL: 📌 Overview llm-d, an open-source distributed inference stack, hits 1.0. It turns vLLM and SGLang servers on Kubernetes into one serving pool. It is a CNCF Sandbox project with 500+ contributors. ❓ The problem Agents share long prefixes and arrive in bursts, causing redundant computation, memory pressure, and interference with interactive users. 🛠️ Key mechanisms ・Cache-aware routing with priority-based flow control ・KV cache offload to host memory, SSD, and Managed Lustre ・Disaggregated prefill/decode with safe rolling updates via DisaggregatedSet ・Time-slicing that fills idle RL capacity with async inference 📊 Results (vendor-reported) ・Agentic coding on TPU: median TTFT down 85%+, throughput 2.4x ・Cache hit rate from 35% to 70% ・About 60% fewer GPU-hours on A3 Mega ・RL duty cycle from ~40% to 70%+ 🌍 Use cases Snap, ZenteiQ, and Mistral use it. You can start by putting prefix-aware routing in front of an existing vLLM deployment. #llmd# #LLMInference#

llm-d flow control, chapter 3: where control starts. Throughput can plateau while latency keeps rising. The admission point is where flow control decides to keep a request in the queue or admit it to the model server pod. Using a load test, you can find that cutoff point for your model and workload.

llm-d flow control, chapter 2: shared inference under burst pressure. A GPU pool can have spare capacity on average and still run out during a traffic burst. With @_llm_d_ flow control enabled, excess requests are queued until capacity becomes available.

llm-d flow control interactive learner, chapter 1: capacity utilization. Here’s the same traffic in separate tenant-GPU reservations versus a shared pool. Sharing lets tenants use capacity that would otherwise sit idle.

The @vllm_project & llm-d maintainers at @RedHat_AI are some of the hardest-working and leading experts on inference in the world 🚀 Everyone in the ML community can tell you how kind & helpful @robertshaw21 & @mgoin_ are. We’re grateful to occasionally collaborate with them. ❤️

Scale distributed inference and control AI serving costs with confidence. llm-d 1.0 brings multi-tenant quality of service, zero-downtime rolling updates, and prefix-cache-aware routing to your production workloads. Explore the release: #PyTorchCon#

Run the open inference stack the way you built it. On AKS, vLLM, KServe, llm-d, and Ray work as they do upstream, and KAITO makes getting a model into production a Kubernetes operation. See it live at #KubeCon#: