vLLM hit new peak bs=1 decode on Kimi-K3: 464 tok/s 🚀 Under a low-entropy reasoning workload, Kimi-K3 + DSpark on vLLM reaches 464 tok/s on batch size 1 with 4×4 GB300. This benchmark is fully reproducible with public image: vllm/vllm-openai:kimi-k3 and @inferact's DSpark draft model linked in the thread. 1/3

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to @Kimi_Moonshot AI for the model release and partnership, @inferact for leading the vLLM optimizations, and to our partners at @nvidia, @AMD, and the broader vLLM community. 1/6

🎉 Congrats to @poolsideai on Laguna S 2.1, a new open-weight model built for agentic coding and long-horizon work. 🧠 118B sparse MoE, only 8B active per token, up to 1M context, thinking + no-thinking modes, OpenMDW-1.1 🔁 Built to stay on task across long, multi-step runs: plan, call tools, check its work, recover, keep going 🖥️ The official NVFP4 quant runs locally on a single @NVIDIAAI DGX Spark This model is a scale up of the Laguna XS 2.1 architecture and vLLM runs it out of the box. Keep your existing Laguna serve setup, and the poolside_v1 tool-call and reasoning parsers already work.

Great writeup from @khluu000 👏 How does vLLM stay production-quality while merging ~2,000 commits/month and shipping every 2 weeks? The team broke down the three layers that make it possible. A huge community effort—thank you to everyone who is helping along the way.

Congrats to @Kimi_Moonshot on the Kimi K3 announcement! 🎉 Grateful for the shoutout and collab. The Kimi team announced they contributed a KDA prefix caching implementation directly to vLLM, to be released alongside the model. 🚀 KDA breaks assumptions behind conventional prefix caching, and this upstream work means the community gets efficient long-context serving from day 0. vLLM will support Kimi K3 on day-0 release. Open weights by July 27, 2026.

🎉 Congrats to @PrimeIntellect on Verifiers v1! Its training rollouts run on vLLM for exact token IDs and logprobs, no tokenization drift, keeping rollouts and training in sync. vLLM powers a growing set of open RL infra, prime-rl and others, and it's an area we're going deep on. 🚀

🎙️ @Alibaba_Qwen's Qwen3-Omni listens, reasons, and talks back. Serving that in real time is a pipeline problem, not a single model: a multimodal Thinker, then Talker → Code2Wav for the speech. Each stage bottlenecks differently, so the wins come from optimizing them layer by layer. One neat trick: under load, replicate only the two speech stages and let the heavy multimodal Thinker run once. At high concurrency that lands first audio in ~0.6s instead of ~6s, speech faster than real time, and ~5.4x the throughput on the same GPUs. Built with @AntGroup's Super Computing Technology (SCT) team and the vLLM-Omni team. The blog breaks down the full stack, one bottleneck at a time 👇 🔗

GLM 5.2 DSpark update! The full Speculators training run is well underway and we have the epoch-1 checkpoint ready for your GPUs using vLLM nightly: This improves upon the speedup from the preview checkpoint by another 1.5-2x. Stay tuned for more!

5x lower token costs on DeepSeek V4 in one month! Highlighting the vLLM community at work: day-zero recipes, then relentless optimization across kernels, scheduling, and serving. Every PR counts. 🚀

And it's not locked to DeepSeek's checkpoints. 🧩 The Speculators library ( lets you train and package DSpark draft models in a standard, HF-compatible format that vLLM loads directly. Already validated on Qwen3-8B and GLM-5.2. Run it on vLLM nightly now: vllm serve zai-org/GLM-5.2-FP8 -tp 8 --speculative-config '{"method":"dspark","model":"RedHatAI/GLM-5.2-speculator.dspark-preview","num_speculative_tokens":7,"attention_backend":"FLASH_ATTN","draft_sample_method":"greedy"}' speculators support PR: Thanks to the @RedHat_AI team for the Speculators integration! 🙏

🚀 @deepseek_ai's DSpark speculative decoding now runs natively in vLLM! What it is: a semi-autoregressive drafter that proposes several tokens in parallel with non-causal sliding-window attention, then verifies them in a single pass. Output stays identical, decoding takes fewer steps. How vLLM runs it: it reuses the existing SparseMLA backends instead of custom attention kernels, captures the full draft backbone and sampling loop in one CUDA graph, and works with prefix caching and FP8 KV cache. Performance on DeepSeek-V4-Pro-DSpark (verified on NVIDIA 8×B300 GPUs): - ~250 tokens/s at batch size 1 - average acceptance length ~5 - 12-42% higher acceptance than MTP across draft depths Run with vLLM nightly today: vllm serve deepseek-ai/DeepSeek-V4-Pro-DSpark -tp 8 --trust-remote-code --kv-cache-dtype fp8 --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}' DSpark Core PR: Thanks @deepseek_ai for open-sourcing DSpark, and to @NVIDIAAI and the vLLM community for landing it! 🙏

🚀 Qwen3.6-27B-NVFP4 is inference ready with vLLM on NVIDIA Blackwell GPUs. This checkpoint is optimized for Blackwell and reduces GPU memory requirements by ~2.5x for local AI with open-source models. 🧠 27B params, Hybrid Attention 📊 NVFP4 evals: 86.3 on MMLU Pro, 85.5 on GPQA Diamond 🛠️ Exclusively supported on vLLM as the runtime engine Get started from the Hugging Face checkpoint:

The very first vLLM meetup is at @a16z. Our reboot meetup series in the Bay Area will resume at @a16z in few weeks. Looking forward to see everyone again! @CrusoeAI and @inferact will dive in depth re:production vLLM!

🥂 Join us for a happy hour during @aiDotEngineer World's Fair this Thursday 7/2 in San Francisco, co-hosted with @inferact and @novita_labs! Casual conversation on open weight models, inference, and infra with fellow builders, founders, and researchers. 📍Hosted at the @inferact office , no conference ticket required! RSVP here: