If agents leave GPUs idle while waiting on tools, a scheduler should look at when those tools return. TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving 🔎 Overview A scheduler for agents that alternate between LLM inference and tool calls. It scores tool-return timing together with KV-cache reuse, then picks which sessions to admit via combinatorial optimization. ⚠️ The problem ・Existing schedulers don't use tool-return timing directly ・Greedy per-session decisions can block a better combination of sessions competing for the same capacity 🛠 Method ・Each session gets three values compared in order: time-decayed cache credit (10 s half-life), tool-return readiness, and token count ・A capacity-state dynamic program jointly selects waiting sessions and assigns nodes ・The DP runs in an async worker and is revalidated against live state before applying; overhead is under 0.12% of runtime 📊 Results ・On DeepSeek-V4 Flash, SWE-Lite, 64 workers, vs. the best baseline: up to 26.8% lower latency, 36.3% lower TTFT, 29.5% higher decode throughput ・Swapping the DP for BFD drops decode from 1417.97 to 1233.72 tok/s ・It doesn't win everywhere: TTFT gets worse on AutomationBench at 128 workers #LLMServing# #AIAgents#

6.12 billion requests, 9,174 models, one full year of unsampled production traces — the most comprehensive study of LLM serving workload behavior published to date. A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing 🔍 Overview A year of request-level production traces (April 2025 – April 2026) from Chutes, covering 6.12B requests, 314K users, and 9,174 models. For the first time, the full unsampled picture of LLM serving behavior — including prefix cache reuse and per-instance load — is available for analysis and replay. ⚠️ The problem being solved Existing workload studies are short, sampled, or single-model, making it impossible to evaluate serving system designs under production-representative conditions. This paper closes that gap with a trace researchers can replay directly. 🔬 Key findings · Workloads are non-stationary: request count and actual serving cost follow different trends — short windows are insufficient for capacity planning · Output tokens trend shorter: from hundreds early in the trace to below 100 by year-end · 99% of cache reuse arrives within 15 minutes of the previous request (80% within 0.1 seconds) · LRU matches or beats complex algorithms in most cases; ARC substantially underperforms at intermediate cache sizes · Cache-first routing substantially outperforms round-robin and load-first, with load imbalance remaining within 5–7% · Sticky routing (user pinning) achieves the highest hit ratios but with orders-of-magnitude worse load imbalance — not practical 📊 Experimental results A 100K-token MiniMax-M2.5 request occupies ~27GB of KV state, requiring ~20.4GB of network transfer per cross-instance move. Routing volatility directly induces KV replication, creating a fundamental tradeoff between cache locality and load balance. Cache-first preserves locality while keeping imbalance low because the workload contains many single-turn sessions. #LLM# #MLSystems#

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with @RedHat and taught by @cedricclyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

We’re excited to welcome Mooncake to the PyTorch Ecosystem! Mooncake is designed to solve the “memory wall” in LLM serving. By integrating Mooncake’s high performance KVCache transfer and storage capabilities with PyTorch native inference engines like SGLang, vLLM, and TensorRT-LLM, it unlocks new levels of throughput and scalability for large language model deployments. Mooncake enables prefill decode disaggregation, global KVCache reuse, elastic expert parallelism, and serves as a fault tolerant PyTorch distributed backend. 🔗 #PyTorch# #OpenSourceAI# #LLM# #AIInfrastructure#

💾 TL;DR: reuse a KV cache computed on another node's GPU straight through CXL memory, up to 36.6x faster. Seagate proposes a Kubernetes-native way to share CXL memory across the cluster. Title: Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving URL: 📌 Highlights 🧩 A Kubernetes DRA driver turns CXL regions into schedulable cluster resources 🗂 No Redis/etcd needed, KV slots are tracked inside the region itself 🚀 Cross-node reuse hits 36.6x speedup over recompute at 32K-token prefixes 🎯 Hit rates reach 95.4-99.5% ⚖️ Cross-node vs same-node latency differs by only 1-4% 🐢 Still 1.15-1.70x slower than an on-GPU VRAM prefix cache hit 🔧 Measured 5.8 GB/s out of a 27 GB/s fabric ceiling, room to grow with zero-copy DMA For serving that needs to reuse long prefixes across GPU nodes, this feels like a genuinely practical option. #LLMInference# #Kubernetes#

🌐 Microsoft brought 11 papers to NSDI 2026, the frontier of large-scale networked systems. This digest spans concrete advances for AI infrastructure: 4x throughput from KV-cache reuse, 3.2x over RDMA with CXL memory, and more. Title: Microsoft at NSDI 2026: Advances in large-scale networked systems URL: 📝 Overview NSDI is a leading venue for sharing new research on the design and operation of cloud, AI, and distributed application infrastructure. This post highlights Microsoft's 11 accepted papers spanning datacenter networks, wide-area networking, AI systems, and cloud infrastructure. ❓ Challenges Solved As AI workloads surge, scaling datacenter network, memory, and CPU resources while preserving reliability and cost efficiency keeps getting harder. The papers tackle that frontier from both production operation and research angles. 💡 Key Research Contributions ・DroidSpeak reuses KV caches across same-architecture language model variants for up to 4x throughput with minimal impact on output quality ・SONiC DASH SmartSwitch redesigns cloud network offloading and is deployed at scale in Azure, significantly improving power and space efficiency (community award winner) ・Octopus, a CXL disaggregated memory pod, achieves RPCs 3.2x faster than in-rack RDMA and 2.4x faster than CXL switches on three-server prototypes ・HarvestContainers dynamically harvests cores from latency-sensitive containers, using up to 75% of spare CPU while keeping tail latency within 4% of standalone 🎯 Use Cases It informs design decisions for faster LLM serving, memory disaggregation, container resource utilization, and network-protocol verification across cloud and AI infrastructure. 📊 Other Results ・Eywa used LLM-based model generation to find 33 bugs (16 previously unknown) in protocol implementations ・ForestColl generates theoretically optimal collective-communication schedules in polynomial time ・AVA is a video-analytics benchmark reaching 75.8% accuracy on 10+ hour videos ・KRAKENGUARD provides fine-grained eBPF isolation for multi-tenant security #NetworkedSystems# #AIInfrastructure#