Introducing SGLang Low-Latency Mode, built with @NVIDIAAI and @AMD. On DeepSeek-V4.1-Flash with DSpark, it delivers up to 1.38× P90 per-user token speed on NVIDIA GB300 (671 to 928 tok/s) and 1.56× on AMD MI355X (332 to 516 tok/s) in @SemiAnalysis_'s AgentX trace replay. Highlights: - NVIDIA: overlapping kernels cut decoder layer time from 96.8 to 68.4 µs - AMD: 2 persistent kernels per layer replace dozens of small ones, with cross-GPU communication overlapped inside them. - New plugin system: custom kernels plug in from external libraries, with automatic fallback. - Low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress and will plug in next.

SGLang brings optimized inference to @nvidia Vera Rubin, with performance gains across attention, MoE, and speculative verification for Kimi K3 NVFP4. Miles by @radixark takes this further with end-to-end RL training, using SGLang for rollouts and the Vera CPU for concurrent agent sandboxes. The teams share their early results, benchmarks, and the engineering behind them 👇

Meet the SGLang Summit 2026 speakers. 🗓️ Nov 12–13 · 📍 Fort Mason, San Francisco 🎟️ Early bird pricing ends Oct 20: Two days to connect with the people building frontier AI infrastructure, from silicon to models and apps. Main stage keynotes, plus breakout deep dives and hands-on labs you can only catch in person. Featuring @LipBuTan1 (Intel), @lilianweng (OpenAI), @soumithchintala (Thinking Machines), @AravSrinivas (Perplexity), Bill Jia (Google), @ctnzr (NVIDIA), @luisceze (NVIDIA), Ujval Kapasi (NVIDIA), @ibab (River AI), @ericzelikman (humans&), @tachim (Harmonic), @marksaroufim (Core Automation), @tydsh (Recursive), Qi Jin (Cerebras), @cHHillee (Thinking Machines), @SimonShaoleiDu (Apodex), @zianglih (humans&), @hsu_byron (Periodic Labs), @JiantaoJ (NVIDIA), Vinay Kumar (DigitalOcean), @lm_zheng (Meta), @ying11231 (RadixArk), @BanghuaZ (RadixArk) and more.

🚀 New blog: Scaling JEV-like decision models with SGLang Decision models need a score, not prose. Classification, ranking, and agent action selection all ask the same thing: which option wins? Open-Jev, for example, scores each candidate separately with a Yes/No prompt. Serving this well raises two issues: Generate + top-k logprobs can drop the label you need, and the shared context can be recomputed for every candidate. SGLang addresses both: - /v1/score returns scores for the exact labels you request (Yes/No, A/B/C) - Multi-item scoring (MIS) computes the shared context once and keeps each candidate isolated - MIS latency stays nearly flat from 2 to 16 candidates, with 16-candidate p95 on Qwen3-8B dropping from 54.1 ms (Generate) to 20.6 ms (MIS) - MIS p95 stays under ~100 ms as load rises on Qwen3-0.6B, vs. seconds for Generate and SIS Huge thanks to the @LinkedIn team for contributing! Benchmarks and launch commands in the blog 👇

SGLang v0.5.20 landed! Welcome @intel XPU to join standard SGLang releases 🎉 Some of our favorite updates: - RL sampling masks make rollouts more reliable, with up to 52% faster decode - Unified Radix Tree adds SWA branching-point caching: ~20pt higher cache hit rate, ~1/3 lower TTFT - DSpark now supports PD + DCP for long-context serving - SGLang Simulator brings scheduler & cache experiments to CPU - ROCm model loading is up to 12.5× faster - SGLang-Diffusion gets up to ~38% lower E2E latency New models include GLM-5.3-Flash, Qwen3.8-Flash-Next, K2 Horizon, Hy4-Preview, FastH3, VDN-H3, and more. Full release notes👇

🚀New blog: Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache 4-bit KV cache is here! NVFP4 KV in SGLang packs ~1.78× more context into GPU memory and speeds up long-context decode by up to 78%. Together with @Alibaba_Qwen and @nvidia, we brought NVFP4 KV cache to Blackwell: - NVFP4 stores KV in just ~56% of FP8's footprint per token -️ Decode throughput jumps +37% / +58% / +78% at 32K / 160K / 1M context - Near-lossless accuracy: matches FP8 on GPQA-Diamond & AIME 2025 (Qwen3.5-397B-A17B) - Higher cache hit rate keeps AgentX throughput scaling where FP8 drops off The whole recipe combines NVFP4 two-level scaling, in-kernel dequantization on decode, and paged KV cache, and you can enable it in SGLang with a single flag --kv-cache-dtype nvfp4

🚀 New Blog: Running DeepSeek-V4-Flash and Kimi-K3 on Consumer Hardware with SSD Expert Pack These models are far too large for a typical PC's memory. WiCi AI and the SGLang team built SSD Expert Pack: routed experts stay on an NVMe SSD, and the runtime loads only the experts the router selects into a GPU cache. On one RTX 5090, 32 GB RAM, and a 2 TB SSD 🔸 DeepSeek-V4-Flash MXFP4: 1.85–1.99 tokens/sec decode 🔸 Kimi-K3 community Q2_K (text-only): ~0.29 tokens/sec decode Thanks to the WiCi AI team for the collaboration! Read the full blog below 👇

Very excited to see native vision come to the @Zai_org GLM family, and impressed by the architecture work behind it! With SGLang features like RadixAttention and HiCache, GLM-5.3-Flash is one of the most efficient and capable choices for agentic work across coding and professional tasks. Can't wait to see what people will build and run with them👏

SGLang brings Day-0 support for Qwen 3.8-Flash-Next, an early preview of the Qwen4 architecture! In collaboration with @Alibaba_Qwen, @NVIDIAAI , and @AIatAMD, we've optimized this 125B MoE (6B active, plus 51B N-gram embeddings) for efficient inference: 1️⃣540 tok/s decode speed: Achieved at BS=1 on NVIDIA B200 (TP4) using the day-0 NVFP4 checkpoint provided by @radixark and IndexShare MTP. 2️⃣GDN + QSA Hybrid: 36 Gated DeltaNet layers + 12 Qwen Sparse Attention layers drastically cut long-context compute and KV overhead. 3️⃣N-Gram Host Offloading: Offloads 51.2B embedding params to host memory—saving 23.5 GiB VRAM/GPU (TP4) and boosting KV capacity by +78.5% with zero throughput drop. 4️⃣HyperConnection Kernels: Fused Mix/Combine operators (via FlashInfer) deliver a 2.05× kernel speedup and a 7.6% end-to-end boost. 5️⃣IndexShare MTP: Freezes and reuses target QSA index selections across draft steps, eliminating draft indexer bottlenecks. 📦 Model Checkpoint: 📖 Read the full blog & cookbook: #SGLang# #Qwen# #NVIDIA# #AMD# #OpenSource# #LLM#

Today, we’re introducing the Weight Cache Daemon for SGLang. 🚀 On Ling-2.6-1T FP8, it reduced weight loading to ~0.63s, up to ~780× faster than disk loading, and cut total engine startup from 8.8 minutes to ~0.53 minutes. Here’s how it works.

🚀 New blog: Pushing the Limits of Serving DeepSeek-V4-Pro DeepSeek-V4-Pro (1.6T MoE) on H20 reaches 271 output tokens/s at batch size 1, just 1.42× off B300 on hardware with no native FP4 Tensor Cores. Together with @ant_oss, we built a scenario-specific serving stack on SGLang: - 74.8%–78.0% peak TPOT reduction at batch size 1 from optimized DSpark - 1M-token prefill in 43.7s, 36.5% geomean prefill throughput gain - 10.14× full-token KV capacity from Humming MXFP4AFP8 + Online C128 - 2.20× per-GPU decode throughput at 4K (319.9 to 703.2 tok/s/GPU)

SGLang now has Day 0 support for MiniMax-Music 3 from @MiniMax_AI 🎵 A music generation model powered by a Qwen3 + RVQ autoregressive backbone and a flow-matching DIT/DAV decoder. POV: you’re parkouring through old Shanghai alleys while Music 3 scores your main-character moment 🎵🧡 Visuals by @Hailuo_AI, music by @MiniMax_AI MiniMax-Music 3. Make your own with SGLang Omni Cookbook👇

Congrats to the @deepseek_ai team on launching DeepSeek-V4-Pro-0813 and DeepSeek harness🐳 The new checkpoint just dropped: Harness repo: Another impressive jump from the previews, and this is a serious model for agentic coding and long horizon tasks 🤯 And the built-in DSpark draft module keeps generation fast even under high concurrency. SGLang is the best engine to run the DeepSeek V4 family, and every optimization we've already landed carries straight over to this release. Cookbook 👇

🎉 Day 0 support for @nvidia Nemotron 3.5 Lightning is live in SGLang! A 30B hybrid Mamba-Transformer MoE with 3B active params, distilled from Nemotron 3 Ultra for always-on agents: ✅ Three built-in speculators: MTP, DFlash, DSpark ✅ Trained for agent harnesses: coding, tool use, multi-turn ✅ Up to 1M-token context, BF16 + NVFP4 available ✅ Runs on Jetson, DGX Spark, Hopper and Blackwell architectures Run it now with SGLang!

🎉 Day-0 support for Ling-3.0-flash from @AntLingAGI is now live in SGLang! A 124B MoE model built for production agents with: > Hybrid-linear from step 0 of pretraining: KDA + MLA stacked 5:1, 1/64 sparse MoE > 10,000+ interactive training environments > New INT4 and MXFP4 variants, running end-to-end on a single NVIDIA DGX Spark via the Spark-adapted SGLang path ⭐️ What makes long agent runs fast: Ling-3.0-flash natively integrates SGLang HiCache + Mooncake hierarchical caching, cutting TTFT by 60% to over 80% on long inputs. Try it in your agent stack today!