LMSYS Org(@lmsysorg):SGLang brings Day-0 support for Qwen 3.8-Flash-Next, an early preview of the Qwen4 architecture! In collaboration with @Alibaba_Qwen, @NVIDIAAI , and @AIatAMD, we've optimized this 125B MoE (6B active, plus 51B N-gram embeddings) for efficient inference: 1️⃣540 tok/s decode speed: Achieved at BS=1 on NVIDIA B200 (TP4) using the day-0 NVFP4 checkpoint provided by @radixark and IndexShare MTP. 2️⃣GDN + QSA Hybrid: 36 Gated DeltaNet layers + 12 Qwen Sparse Attention layers drastically cut long-context compute and KV overhead. 3️⃣N-Gram Host Offloading: Offloads 51.2B embedding params to host memory—saving 23.5 GiB VRAM/GPU (TP4) and boosting KV capacity by +78.5% with zero throughput drop. 4️⃣HyperConnection Kernels: Fused Mix/Combine operators (via FlashInfer) deliver a 2.05× kernel speedup and a 7.6% end-to-end boost. 5️⃣IndexShare MTP: Freezes and reuses target QSA index selections across draft steps, eliminating draft indexer bottlenecks. 📦 Model Checkpoint: https://t.co/P9UNGGGVnh 📖 Read the full blog & cookbook: https://t.co/Oxq87Sl9S9 #SGLang #Qwen #NVIDIA #AMD #OpenSource #LLM

2026.08.26 15:16

SGLang brings Day-0 support for Qwen 3.8-Flash-Next, an early preview of the Qwen4 architecture! In collaboration with @Alibaba_Qwen, @NVIDIAAI , and @AIatAMD, we've optimized this 125B MoE (6B active, plus 51B N-gram embeddings) for efficient inference: 1️⃣540 tok/s decode speed: Achieved at BS=1 on NVIDIA B200 (TP4) using the day-0 NVFP4 checkpoint provided by @radixark and IndexShare MTP. 2️⃣GDN + QSA Hybrid: 36 Gated DeltaNet layers + 12 Qwen Sparse Attention layers drastically cut long-context compute and KV overhead. 3️⃣N-Gram Host Offloading: Offloads 51.2B embedding params to host memory—saving 23.5 GiB VRAM/GPU (TP4) and boosting KV capacity by +78.5% with zero throughput drop. 4️⃣HyperConnection Kernels: Fused Mix/Combine operators (via FlashInfer) deliver a 2.05× kernel speedup and a 7.6% end-to-end boost. 5️⃣IndexShare MTP: Freezes and reuses target QSA index selections across draft steps, eliminating draft indexer bottlenecks. 📦 Model Checkpoint: 📖 Read the full blog & cookbook: #SGLang# #Qwen# #NVIDIA# #AMD# #OpenSource# #LLM#

SGLang@sgl_project

2026.08.26 12:42

Congrats to @Alibaba_Qwen on launching Qwen3.8-Flash! SGLang is proud to be a day-0 partner supporting the new architecture preview for Qwen4. It's a 125B main model with 51B of N-gram embeddings and 6B activated per token. The 51B N-gram embeddings scale model capacity with almost no extra compute per token, and can sit in host memory with async prefetch instead of occupying GPU memory. The GDN + QSA hybrid attention gives you efficient memory and precise retrieval at the same time on long-horizon tasks, while Gated Residual gives the model 4 lanes instead of 1 to pass information between layers. And it's trained with Muon! We're excited for what's next with Qwen4, and we already have plenty of ideas for how to use the N-gram embeddings in new deployment setups. Stay tuned! Blog and cookbook in the comments👇

Forward to community