6.12 billion requests, 9,174 models, one full year of unsampled production traces — the most comprehensive study of LLM serving workload behavior published to date. A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing 🔍 Overview A year of request-level production traces (April 2025 – April 2026) from Chutes, covering 6.12B requests, 314K users, and 9,174 models. For the first time, the full unsampled picture of LLM serving behavior — including prefix cache reuse and per-instance load — is available for analysis and replay. ⚠️ The problem being solved Existing workload studies are short, sampled, or single-model, making it impossible to evaluate serving system designs under production-representative conditions. This paper closes that gap with a trace researchers can replay directly. 🔬 Key findings · Workloads are non-stationary: request count and actual serving cost follow different trends — short windows are insufficient for capacity planning · Output tokens trend shorter: from hundreds early in the trace to below 100 by year-end · 99% of cache reuse arrives within 15 minutes of the previous request (80% within 0.1 seconds) · LRU matches or beats complex algorithms in most cases; ARC substantially underperforms at intermediate cache sizes · Cache-first routing substantially outperforms round-robin and load-first, with load imbalance remaining within 5–7% · Sticky routing (user pinning) achieves the highest hit ratios but with orders-of-magnitude worse load imbalance — not practical 📊 Experimental results A 100K-token MiniMax-M2.5 request occupies ~27GB of KV state, requiring ~20.4GB of network transfer per cross-instance move. Routing volatility directly induces KV replication, creating a fundamental tradeoff between cache locality and load balance. Cache-first preserves locality while keeping imbalance low because the workload contains many single-turn sessions. #LLM# #MLSystems#

👑 Atria Dawn Preview is here, built to complete real research and engineering work. 📜 MIT. 🤖 ⚙️ Built on a 744B MoE foundation with a 256K context window. Standard and FP8 weights are available. 🔬 Discovery workflows cover evidence gathering, deep research, experiment design, execution, analysis, and recovery from failure. 🏆 Leads the reported comparison on AutomationBench, BFCL v4, CyberGym, DeepSearchQA, and BrowseComp. Scores include 53.8, 77.0, 86.5, 96.0, and 92.5 respectively. 🧩 Creation and delivery capabilities span software, interactive apps, ML systems, visualizations, reports, and presentations. 🛡 Cybersecurity support covers analysis, vulnerability validation, remediation, and retesting in authorized environments.

🚀 An ebook that takes FastAPI from your first endpoint all the way to production-scale AI systems — going deep on LLM/RAG serving, with interview questions at the end of every chapter. Title: FastAPI for AI Engineers: From First Endpoint to Production-Scale AI Systems URL: 🚀 Overview A practical guide (First Edition, 2026, by AI Engineering Insider) for AI engineers serving ML models and LLM/RAG systems in production with Python. It spans 10 chapters and 100 interview questions, woven with real-incident case studies and cost-model sidebars. ❓ Challenges Solved ・Building a model is one skill; serving it safely as a scalable production API is another ・LLM/RAG serving has its own hard parts — streaming, guardrails, cost control The book reframes FastAPI as "the de facto serving layer for AI and ML systems." 💡 Structure & Tech Covered ・Foundations: ASGI/WSGI, Uvicorn, OpenAPI, and Pydantic v2 schema separation and validation ・Implementation: idempotency, meaningful status codes, pagination, and the Router→Service→Repository clean architecture ・DB/Security: SQLAlchemy/SQLModel/Alembic, N+1, pool sizing, JWT, BOLA defense, OWASP API Top 10 ・Async: "never block the event loop," def vs async def, and httpx retries/circuit breakers 🎯 The Core (Ch.9: AI/RAG/LLM) ・Load model weights once in lifespan; offload CPU inference to a thread ・An LLM gateway centralizing auth, prompts, guardrails, and cost metering, with SSE token streaming ・Build RAG with embeddings + a vector DB (start with pgvector); validate output with Pydantic, then retry on failure ・Enforce max_tokens as a "spending limit" via the type system 📊 Highlights ・Practice-first, learning from real incidents (Netflix, Stripe, GitLab, Optus, Air Canada) ・Ch.10 covers Gunicorn+Uvicorn, K8s liveness/readiness, the three pillars of observability (p99 vs p50), and SLO-based alerting #FastAPI# #AIEngineering#

An OpenAI friend told me he burns 300M GPT-5.5 tokens/day. The top one in his team burns billions of tokens/day. Codex coding for them every night. Databricks also gives engineers unlimited tokens. We're looking for cracked inference engineers to join us at Databricks AI to produce trillions of tokens, insanely fast. DM me if you have: - Contributed to open-source ML systems like SGLang/vLLM/PyTorch - Experience serving LLMs at large scale Databricks AI runs like a startup. Lots of exciting things to build!