🚀 New blog: Scaling JEV-like decision models with SGLang Decision models need a score, not prose. Classification, ranking, and agent action selection all ask the same thing: which option wins? Open-Jev, for example, scores each candidate separately with a Yes/No prompt. Serving this well raises two issues: Generate + top-k logprobs can drop the label you need, and the shared context can be recomputed for every candidate. SGLang addresses both: - /v1/score returns scores for the exact labels you request (Yes/No, A/B/C) - Multi-item scoring (MIS) computes the shared context once and keeps each candidate isolated - MIS latency stays nearly flat from 2 to 16 candidates, with 16-candidate p95 on Qwen3-8B dropping from 54.1 ms (Generate) to 20.6 ms (MIS) - MIS p95 stays under ~100 ms as load rises on Qwen3-0.6B, vs. seconds for Generate and SIS Huge thanks to the @LinkedIn team for contributing! Benchmarks and launch commands in the blog 👇

SGLang v0.5.20 landed! Welcome @intel XPU to join standard SGLang releases 🎉 Some of our favorite updates: - RL sampling masks make rollouts more reliable, with up to 52% faster decode - Unified Radix Tree adds SWA branching-point caching: ~20pt higher cache hit rate, ~1/3 lower TTFT - DSpark now supports PD + DCP for long-context serving - SGLang Simulator brings scheduler & cache experiments to CPU - ROCm model loading is up to 12.5× faster - SGLang-Diffusion gets up to ~38% lower E2E latency New models include GLM-5.3-Flash, Qwen3.8-Flash-Next, K2 Horizon, Hy4-Preview, FastH3, VDN-H3, and more. Full release notes👇

🚀New blog: Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache 4-bit KV cache is here! NVFP4 KV in SGLang packs ~1.78× more context into GPU memory and speeds up long-context decode by up to 78%. Together with @Alibaba_Qwen and @nvidia, we brought NVFP4 KV cache to Blackwell: - NVFP4 stores KV in just ~56% of FP8's footprint per token -️ Decode throughput jumps +37% / +58% / +78% at 32K / 160K / 1M context - Near-lossless accuracy: matches FP8 on GPQA-Diamond & AIME 2025 (Qwen3.5-397B-A17B) - Higher cache hit rate keeps AgentX throughput scaling where FP8 drops off The whole recipe combines NVFP4 two-level scaling, in-kernel dequantization on decode, and paged KV cache, and you can enable it in SGLang with a single flag --kv-cache-dtype nvfp4

🚀 New Blog: Running DeepSeek-V4-Flash and Kimi-K3 on Consumer Hardware with SSD Expert Pack These models are far too large for a typical PC's memory. WiCi AI and the SGLang team built SSD Expert Pack: routed experts stay on an NVMe SSD, and the runtime loads only the experts the router selects into a GPU cache. On one RTX 5090, 32 GB RAM, and a 2 TB SSD 🔸 DeepSeek-V4-Flash MXFP4: 1.85–1.99 tokens/sec decode 🔸 Kimi-K3 community Q2_K (text-only): ~0.29 tokens/sec decode Thanks to the WiCi AI team for the collaboration! Read the full blog below 👇

Very excited to see native vision come to the @Zai_org GLM family, and impressed by the architecture work behind it! With SGLang features like RadixAttention and HiCache, GLM-5.3-Flash is one of the most efficient and capable choices for agentic work across coding and professional tasks. Can't wait to see what people will build and run with them👏

SGLang brings Day-0 support for Qwen 3.8-Flash-Next, an early preview of the Qwen4 architecture! In collaboration with @Alibaba_Qwen, @NVIDIAAI , and @AIatAMD, we've optimized this 125B MoE (6B active, plus 51B N-gram embeddings) for efficient inference: 1️⃣540 tok/s decode speed: Achieved at BS=1 on NVIDIA B200 (TP4) using the day-0 NVFP4 checkpoint provided by @radixark and IndexShare MTP. 2️⃣GDN + QSA Hybrid: 36 Gated DeltaNet layers + 12 Qwen Sparse Attention layers drastically cut long-context compute and KV overhead. 3️⃣N-Gram Host Offloading: Offloads 51.2B embedding params to host memory—saving 23.5 GiB VRAM/GPU (TP4) and boosting KV capacity by +78.5% with zero throughput drop. 4️⃣HyperConnection Kernels: Fused Mix/Combine operators (via FlashInfer) deliver a 2.05× kernel speedup and a 7.6% end-to-end boost. 5️⃣IndexShare MTP: Freezes and reuses target QSA index selections across draft steps, eliminating draft indexer bottlenecks. 📦 Model Checkpoint: 📖 Read the full blog & cookbook: #SGLang# #Qwen# #NVIDIA# #AMD# #OpenSource# #LLM#

Today, we’re introducing the Weight Cache Daemon for SGLang. 🚀 On Ling-2.6-1T FP8, it reduced weight loading to ~0.63s, up to ~780× faster than disk loading, and cut total engine startup from 8.8 minutes to ~0.53 minutes. Here’s how it works.

🚀 New blog: Pushing the Limits of Serving DeepSeek-V4-Pro DeepSeek-V4-Pro (1.6T MoE) on H20 reaches 271 output tokens/s at batch size 1, just 1.42× off B300 on hardware with no native FP4 Tensor Cores. Together with @ant_oss, we built a scenario-specific serving stack on SGLang: - 74.8%–78.0% peak TPOT reduction at batch size 1 from optimized DSpark - 1M-token prefill in 43.7s, 36.5% geomean prefill throughput gain - 10.14× full-token KV capacity from Humming MXFP4AFP8 + Online C128 - 2.20× per-GPU decode throughput at 4K (319.9 to 703.2 tok/s/GPU)

SGLang now has Day 0 support for MiniMax-Music 3 from @MiniMax_AI 🎵 A music generation model powered by a Qwen3 + RVQ autoregressive backbone and a flow-matching DIT/DAV decoder. POV: you’re parkouring through old Shanghai alleys while Music 3 scores your main-character moment 🎵🧡 Visuals by @Hailuo_AI, music by @MiniMax_AI MiniMax-Music 3. Make your own with SGLang Omni Cookbook👇

Congrats to the @deepseek_ai team on launching DeepSeek-V4-Pro-0813 and DeepSeek harness🐳 The new checkpoint just dropped: Harness repo: Another impressive jump from the previews, and this is a serious model for agentic coding and long horizon tasks 🤯 And the built-in DSpark draft module keeps generation fast even under high concurrency. SGLang is the best engine to run the DeepSeek V4 family, and every optimization we've already landed carries straight over to this release. Cookbook 👇

🎉 Day 0 support for @nvidia Nemotron 3.5 Lightning is live in SGLang! A 30B hybrid Mamba-Transformer MoE with 3B active params, distilled from Nemotron 3 Ultra for always-on agents: ✅ Three built-in speculators: MTP, DFlash, DSpark ✅ Trained for agent harnesses: coding, tool use, multi-turn ✅ Up to 1M-token context, BF16 + NVFP4 available ✅ Runs on Jetson, DGX Spark, Hopper and Blackwell architectures Run it now with SGLang!

🎉 Day-0 support for Ling-3.0-flash from @AntLingAGI is now live in SGLang! A 124B MoE model built for production agents with: > Hybrid-linear from step 0 of pretraining: KDA + MLA stacked 5:1, 1/64 sparse MoE > 10,000+ interactive training environments > New INT4 and MXFP4 variants, running end-to-end on a single NVIDIA DGX Spark via the Spark-adapted SGLang path ⭐️ What makes long agent runs fast: Ling-3.0-flash natively integrates SGLang HiCache + Mooncake hierarchical caching, cutting TTFT by 60% to over 80% on long inputs. Try it in your agent stack today!

Inkling-small is out today! With SGLang, you can get 648 tok/s decode with DSpark (simulated acc len=4) and 288 tok/s w/o DSpark, under the same setup (8x @NVIDIAAI B200, TP 8, NVFP4, bs=1). What makes this model different is the size. 276B total with 12B active is a sweet spot for RL, and both LoRA and full-parameter training become well within reach. Miles is ready and verified for multimodal RL on Inkling-small, so you can turn your multimodal data into real capability gains. At ~1/4 the size, Inkling-small matches the bigger version in capability and even wins on some benchmarks. Run Inkling-small with SGLang, and customize it with Miles.

SGLang day-0 speed on Kimi K3: 423 tok/s (measured on gsm8k), plus RL support ready in Miles @radixark! How the largest open-source model runs this fast: we natively implemented and deeply optimized K3’s new architecture with fused KDA decode kernels, DP attention, DSpark, PD disagg, and KDA-aware prefix caching. We've passed Kimi Vendor Verifier and are ready for production! Thanks to @Kimi_Moonshot, @nvidia, @AMD, @KVCache_AI, @modal, and @baseten for building this with us, and to @googlecloud, @nebiustf, @fal, @digitalocean, @runpod, @DeepInfra and @gmi_cloud for serving K3 on SGLang. Blog, cookbook, benchmarks in the comments. P.S. This demo video? Kimi K3 made it itself. Play the game 👇

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks @deepseek_ai for open-sourcing! Blog with full technical details and commands to run below 👇