SGLang-Diffusion is a high-performance serving framework for diffusion models, designed for both large-scale offline generation and latency-sensitive real-time inference. At PyTorch Conference North America 2026, Yihao Wang and Kevin Mi of @RadixArk will present the design of @sgl_project-Diffusion and the PyTorch-based inference optimizations behind it, including efficient request scheduling, memory management, batching strategies, and execution-path optimizations for diffusion pipelines. Join us in San Jose, October 20-21: #PyTorchCon#

🍎 SGLang's main inference stack now runs on Apple Silicon. Previously, running SGLang on a Mac meant using its separate MLX backend. With v0.5.22, Apple Silicon now gets a standard PyTorch/MPS backend too. That means Macs can now run through SGLang's main pipeline ⚙️ scheduler 🧠 model runner 💾 memory-pool system 🌐 serving stack while MLX remains available as another backend. And we already have a first benchmark. On an M4 Pro with 48GB unified memory, Qwen3-0.6B in an offline batch-4 test reached: ⚡ 65.9 tps median decode 🚀 2,531 tps prefill On an M4 Pro with 48GB unified memory, SGLang reports Qwen3-0.6B at: ⚡ 65.9 tps median decode 🚀 2,531 tps prefill The Mac support also understands Apple's unified-memory limits properly. SGLang now plans memory around the amount Metal will actually allow. Link in ALT.

⚙ SGLang Diffusion Is Retiring torch.compile, With Help From an AI Agent After testing 35 diffusion model presets on H100 and H200 GPUs, SGLang Diffusion found that eager execution now usually matches or outperforms torch.compile. Zhihu contributor BBuf argues that this is more than a benchmark result. Agent-written optimizations and Breakable CUDA Graphs can now replace the two main benefits that torch.compile once provided. 1️⃣ The “seven sins” are three production failures torch.compile can fuse kernels and reduce Python overhead. But in production diffusion serving, BBuf found that its operational cost often outweighed those gains. 🔹 Slow and unpredictable Compilation may take several minutes, and complex models can take more than half an hour. Performance also changes unpredictably across compiler modes, models, and GPU generations. An optimization that works on one GPU may become a regression on another. 🔹 Hostile to manual optimization A custom fused kernel may accelerate eager execution but slow down the compiled model. Graph breaks or broader compiler fusion can erase the improvement. Developers then have to debug both their kernels and the compiler’s changing graph decisions. 🔹 Opaque and difficult to maintain Compiled traces are hard to map back to Python calls. Generated kernels are also difficult to inspect, benchmark independently, or modify. Over time, engineers are pulled into compiler-specific fusion rules and workarounds instead of optimizing the actual model. 2️⃣ The current benchmark changed the calculation In SGLang Diffusion’s latest H100 and H200 tests, torch.compile was usually slower than eager execution or roughly tied with it. Compile jobs taking longer than three minutes were stopped. Several models also encountered timeouts or runtime failures. Some models had previously benefited from compilation. But after explicit kernel optimization, most of those gaps disappeared. For models dominated by launch overhead, eager execution plus Breakable CUDA Graphs could match or beat the compiled path. 3️⃣ An Agent replaced the hidden optimizations The team did not build an elaborate multi-agent system. They used Fable with a simple optimization flow and two Kernel Design Agent skills. The Agent profiled the VAE and model components, identified bottlenecks, implemented explicit optimizations, and repeatedly checked correctness and performance. Within two weeks, this workflow optimized more than 30 diffusion models. Their eager paths reached or surpassed the performance previously provided by torch.compile. The crucial difference is ownership: these optimizations now exist as visible, testable, and maintainable code. 4️⃣ Breakable CUDA Graph replaced the other benefit Kernel fusion is only one reason to use torch.compile. The other is reducing framework overhead through graph execution. SGLang previously relied on compiler-assisted Piecewise CUDA Graphs. After dealing with substantial compiler complexity, it developed Breakable CUDA Graph, which captures reusable execution paths while allowing controlled graph breaks. SGLang Diffusion now uses the same approach. It is easier to integrate into real serving systems and can outperform the compiled path on overhead-heavy models. 5️⃣ The deeper shift is from compiler magic to Agent-owned code A general compiler applies hidden transformations at runtime. An Agent can instead inspect the workload, reproduce useful transformations as source code, and validate them on the target hardware. This makes performance work easier to profile, review, debug, and adapt. BBuf’s claim is specifically about diffusion serving, not every PyTorch workload. But within that scope, the conclusion is blunt: If Agents can reproduce the useful kernel optimizations, while CUDA Graph techniques remove framework overhead, torch.compile no longer has a unique role. 🔗 Full Reading: 🔗 Performance results: 🔗 Kernel Design Agents: 🔗 SGLang: 🔗 Agent optimization skills: #SGLang# #PyTorch# #CUDA# #AIInfra# #DiffusionModels# #AIAgents#

Happy to join the @lmsysorg SGLang meetup at LinkedIn. So much talent and inspiration in this space. If you want to build in this space, come work with us at With @trajrl , we're building the best agent infra. DMs open.

@ornith_ Ornith-1.5-9B python -m sglang.launch_server \ --model-path ornith-ai/Ornith-1.5-9B \ --served-model-name Ornith-1.5-9B \ --host 0.0.0.0 --port 8000 \ --context-length 262144 \ --mem-fraction-static 0.85 \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys @lmsysorg and RadixArk @radixark, and taught by Richard Chen @richardczl, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

Want to hear more about new vLLM and SGLang engines built on a new PyTorch-native TPU backend, TorchTPU? Find Qi Zhou of @Google, Colin Taylor and Angela Yi of @Meta at PyTorch Conference North America in San Jose where they will discuss how they focused on making the existing @sgl_project & @vllm_project infrastructure work natively on TPU, preserving the schedulers, batching systems, OpenAI-compatible APIs, and torch.compile workflows already familiar to GPU users. Because the serving engines remain upstream, new models and features can run on TPU without requiring separate TPU-specific implementations. Learn more about how it works across the stacks & what they learned running this in production, including compile time, device placement under tracing, KV cache layout, and cross-chip collectives. Register for #PyTorchCon# NA now:

🚀 New blog: Scaling JEV-like decision models with SGLang Decision models need a score, not prose. Classification, ranking, and agent action selection all ask the same thing: which option wins? Open-Jev, for example, scores each candidate separately with a Yes/No prompt. Serving this well raises two issues: Generate + top-k logprobs can drop the label you need, and the shared context can be recomputed for every candidate. SGLang addresses both: - /v1/score returns scores for the exact labels you request (Yes/No, A/B/C) - Multi-item scoring (MIS) computes the shared context once and keeps each candidate isolated - MIS latency stays nearly flat from 2 to 16 candidates, with 16-candidate p95 on Qwen3-8B dropping from 54.1 ms (Generate) to 20.6 ms (MIS) - MIS p95 stays under ~100 ms as load rises on Qwen3-0.6B, vs. seconds for Generate and SIS Huge thanks to the @LinkedIn team for contributing! Benchmarks and launch commands in the blog 👇