Kimi K3 on @vllm_project is now 2.2–2.8× faster 🚀 Inferact is proud to have co-led this optimization effort with @RedHat_AI, @NVIDIAAI, and @Huawei, spanning scheduling, KDA state handling, and custom MoE kernels. Read the technical deep dive:

@googlecloud and Inferact are announcing today a partnership to make TPU a first-class citizen in @vllm_project. This partnership puts both teams on one engineering roadmap to bring TPU to the broader open model ecosystem, optimizing vLLM as the agentic production serving engine for TPU: • Production serving features and optimized kernels • A native PyTorch path via TorchTPU • Moving towards day-0 support for frontier model releases We're also launching a community program: shared TPU capacity for open-source contributors, plus dedicated review and design help from the core vLLM maintainers at Inferact. Everything this collaboration produces is open source. Read the full announcement:

On the Day 0 release of DeepSeekv4.1 Flash, NVIDIA vLLM works out of the box with zero issues across all 6 SKUs: H100, H200, B200, B300, GB200, GB300! Amazing work by the NVIDIA & Inferact teams! In comparison, AMD vLLM still does not work on DeepSeekv4.1 Flash, as we will describe below👇️(1/2)🧵

🐳 DeepSeek-V4.1-Flash is out, and vLLM serves it from day 0, verified on NVIDIA and AMD GPUs! 🎉 552B MoE backbone, native vision, 1M context. Built for agents: 8B active while it reads your prompt, 16B while it writes. If you already run DeepSeek-V4 on vLLM, most of this stack will feel familiar: the hyper-connections, the sliding-window plus compressed sparse attention, DSpark drafting, MXFP4 experts. vLLM has carried all of it since V4 landed. Two things are new, and both are worth a look: ✨ Engram: a quarter of the checkpoint is n-gram memory the model looks up instead of computes. 197B parameters of it. ✨ Only four layers write compressed KV now. The rest of the model shares it. Spin it up 👇 🔗

DeepSeek V4 Pro on @vllm_project can be 106x cheaper than Opus 5 on the same @SemiAnalysis_ AgentX workload. We got here with a stack of optimizations, all open in vLLM, and a blog that covers what works and, perhaps more interestingly, what doesn't. This is why I'm excited about open source. Proud of the @inferact team. Try it on your own agents 🚀

New blog is out: vLLM x AgentX: Optimizing for Real-World Agentic Serving. Agent traffic stresses every layer of the serving stack at once. This post walks the full-stack work for optimizing vLLM on Agentic workloads, including the architecture, framework, and runtime optimizations, measured on AgentX, @SemiAnalysis_'s public agentic benchmark. 🧵1/6

We are excited to bring the first open benchmarking of Google's TPUs to the world Running every day, on many models + scenarios $/token is better than B200 and B300 Huge shout-out to Google @inferact and the InferenceX team at SemiAnalysis to this effort that's taken many months

Inferact is taking off 🛫 vLLM is taking off 🛫 Our team is full of superheroes!!! After months of hard work, our major project is finally public! We’re committed to tackling the hardest challenges in AI inference—working closely with model vendors to optimize token quality and with sovereign AI partners to perfect deployment. Even under strict constraints, we can dramatically increase token throughput and deliver substantial economic value. 🤩🤩🤩 If you’re passionate about inference technology, come join us. If you need high-quality tokens, let’s talk. And if you have compute resources, we’d love to collaborate! 😁😁😁

🎬Video generation faster than playback! 🚀MiniMax H3 on vLLM-Omni + FastVideo's FastH3: a complete 10.1s MP4 - video AND synchronized audio - rendered in 8.7s!⚡️ Thanks to @MiniMax_AI for the great Minimax H3 release, the FastVideo team @haoailab for open-sourcing FastH3 and helping on the serving integration, and @NVIDIAAI for the continued sponsorship and joint optimization efforts!

Recently I tried using K3 for Triton kernels and I was surprised by how good it was. The generated kernels are definitely very different from how Codex would have done it. Not a proper experiment but I asked Codex (Sol medium) and K3 (high) to work a the same problem. K3 came out on top by a large margin. @Kimi_Moonshot @KimiDevs Kimi-K3 was served (locally) with @vllm_project by @inferact 😎

Congratulations to @SemiAnalysis_ on the release of AgentX 1.0 🎊, an open-source multi-turn agentic coding benchmark collected from ~$3M of real traces, running on 1000+ chips and ~2MW of continuously operated compute. We are excited to see @vllm_project’s competitive performance on frontier open models: 🔷130,093 tok/s/chip for DeepSeek V4 Pro 🔷 77,079 tok/s/chip for Minimax M3 🔷 12,479 tok/s/chip for Kimi K3. The following thread covers an overview of the work from @vllm_project and @inferact: what we tested, found, and shipped upstream. This work highlights vLLM’s performance on real-world workloads and our committed focus to making vLLM an agentic-first engine. Optimizing AgentX performance meant tackling three major challenges: prefix reuse, efficient long context parallelism, and scaling performance with PD disaggregation. First, long agentic sessions stress prefix caching and KV cache offloading. Modern hybrid models have greatly reduced the required KV cache sizes, and caching every block boundary still saturates the KV cache pool which causes prefix cache thrashing across sessions. The fix was sparse retention: one state per interval-sized segment plus the latest replay boundary (vllm-project/vllm #43447#, #45845#), then preserving shared-prefix boundaries so the interval can go to 0 for agent sessions (#47782#). That gets us >95% hit rate at 14 concurrent requests with contexts to 1M. The bigger structural change in KV cache offloading was making the shared KV pool distributed. With Mooncake Store as a first-class connector, prefill ranks can hit the prefix cache both within a worker and across workers, so we no longer have to trade cache locality against load balance to keep a cluster busy. Session-aware routing (48048) is what lets the router act on it. For a single node deployment, SimpleCPUOffloadConnector has been greatly improved to support all hybrid model architectures and across both CUDA and ROCm platforms. For DeepSeek V4 Pro on ROCm, this implementation gave +81.7% output throughput and 46.6% lower mean e2e latency versus recomputing the prefix. Kimi K3 at 2.8T barely fits on a single node, and squeezing it in leaves almost no headroom for KV cache, which is exactly what a long multi-turn session needs most. Parallelism strategy matters more here than on any other model we tested. TP8/DCP8 tops the K3 agentic frontier across configs, with B300 vLLM peaking at ~12.5k tok/s/chip at ~8 tok/s/user and GB300 NVL72 on Dynamo + vLLM holding the curve out past 200 tok/s/user. K3 also surfaced a routing bug worth pulling: vllm-project/router#194# fixes the router dropping reasoning_content, which hits any reasoning model served behind it. On MiniMax M3, B200 vLLM reaches ~44k tok/s/chip and vLLM leads TRT-LLM on throughput vs p90 TTFT. M3 and Qwen3.5 both shipped day-0. Finally, fully optimized performance requires scaling the deployment with prefill-decode disaggregation and distributed KV cache offloading. Thanks to vLLM’s MultiConnector, this is natively supported with NIXL PD connector + MooncakeStoreConnector. By rate matching to find the ideal prefill-decode ratio, we achieved 4.45x higher throughput at a 60 tok/s interactivity for DeepSeek V4 Pro on GB300 Dynamo compared to B300. Shoutout to @NVIDIAAI, who co-tuned most of these configs with us. Dynamo's router optimizations took AgentX replay time down 23.7% on the vLLM backend, the AIPerf replay harness is what made the traces runnable at all, and NIXL and their kernels sit under each respective performance point. Shoutout to @AIatAMD as well: the team’s AITER sparse-MLA decode selection led to +5.22% AgentX output throughput; the hybrid AITER/native CSA selector led to 1.21–1.76x e2e performance boost, and the team made prebuilt lmcache gfx942/gfx950 wheels and a published Mooncake ROCm wheel. Thank you to @SemiAnalysis_ for building AgentX and for the collaboration throughout. Agentic workloads are what users serve in real production, and benchmarking on real workloads is what improves vLLM and open source inference. Next up: more upstream work and a blog with a full technical deep dive later this week. Stay tuned. 🚀

We're co-hosting a happy hour with @AMD after Day 1 on the vLLM Conference on Tuesday🎉 Our team will be there, the AMD team will be there, and so will the key builders behind @vllm_project and AMD in production. Come chat inference with us. 🍸 Tuesday Aug 25, 6pm, San Francisco.

We're capping Day 1 of the first vLLM Conference with a happy hour hosted by @AMD and @inferact 🎉! We'll have engineers from AMD, Inferact and the vLLM community for casual conversations alongside drinks and food. 📅Tue Aug 25, 6-9pm, San Francisco. No conference ticket needed, RSVP below!

Six weeks ago, serving DSpark in vLLM meant choosing a draft length for your traffic and living with it. Now you set that length once, and vLLM decides how much of the draft to verify every step. ✨ On DeepSeek-V4-Pro-0813, the first token of a 7-token draft survives verification more than 70% of the time. The last one, less than 10%. 📈 One config, adaptive verification on with num_speculative_tokens 7, holds the Pareto frontier from concurrency 1 to 256 on 8×B300. Long draft at low load, short at high load. On main behind enable_adaptive_verification, for DSpark with a confidence head on Flash Attention or DSV4 attention. More backends and models are in bring-up. Thanks to Lucas Wilkinson (@RedHat_AI) and Ben Chislett (@NVIDIAAI), and to @deepseek_ai for DSpark and the varlen DeepGEMM indexer kernel. 🔗

🍰 CAKE paper's out, the design bet: the compiler isn't a fixed black box the agent calls — it's part of the harness, and it's under evolution too. CAKE didn't inherit existing abstraction layer. no tile/layout abstractions: the vocabulary was distilled by agents from a corpus of production kernels. every pattern the agent couldn't express pushed new primitives into the IR, and the analyses to keep them checkable. every barrier/layout bug that kept coming back became a verifier rule. none of this can be designed up front. the IR has to co-evolve with the kernels, and the workload tells you what's missing, the corpus tells you if the fix broke anything. the best language for an agent is the one that tells you what's illegal, what's slow, and which decision might made it faster.