I’ll be at #MLSys# this week, May 18–22 🚀 PyTorch Foundation will have a booth with experts on PyTorch, vLLM, Ray + other foundation projects. Come by, ask questions, and meet the teams building open AI infra 🔥 I’m also speaking Monday morning on agentic self-improvement with OpenRoll 🤖 See you there 👋 #PyTorch# #vLLM# #Ray# @PyTorch @vllm_project @raydistributed @linuxfoundation @aaif_io

6.12 billion requests, 9,174 models, one full year of unsampled production traces — the most comprehensive study of LLM serving workload behavior published to date. A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing 🔍 Overview A year of request-level production traces (April 2025 – April 2026) from Chutes, covering 6.12B requests, 314K users, and 9,174 models. For the first time, the full unsampled picture of LLM serving behavior — including prefix cache reuse and per-instance load — is available for analysis and replay. ⚠️ The problem being solved Existing workload studies are short, sampled, or single-model, making it impossible to evaluate serving system designs under production-representative conditions. This paper closes that gap with a trace researchers can replay directly. 🔬 Key findings · Workloads are non-stationary: request count and actual serving cost follow different trends — short windows are insufficient for capacity planning · Output tokens trend shorter: from hundreds early in the trace to below 100 by year-end · 99% of cache reuse arrives within 15 minutes of the previous request (80% within 0.1 seconds) · LRU matches or beats complex algorithms in most cases; ARC substantially underperforms at intermediate cache sizes · Cache-first routing substantially outperforms round-robin and load-first, with load imbalance remaining within 5–7% · Sticky routing (user pinning) achieves the highest hit ratios but with orders-of-magnitude worse load imbalance — not practical 📊 Experimental results A 100K-token MiniMax-M2.5 request occupies ~27GB of KV state, requiring ~20.4GB of network transfer per cross-instance move. Routing volatility directly induces KV replication, creating a fundamental tradeoff between cache locality and load balance. Cache-first preserves locality while keeping imbalance low because the workload contains many single-turn sessions. #LLM# #MLSystems#