⚙ SGLang Diffusion Is Retiring torch.compile, With Help From an AI Agent After testing 35 diffusion model presets on H100 and H200 GPUs, SGLang Diffusion found that eager execution now usually matches or outperforms torch.compile. Zhihu contributor BBuf argues that this is more than a benchmark result. Agent-written optimizations and Breakable CUDA Graphs can now replace the two main benefits that torch.compile once provided. 1️⃣ The “seven sins” are three production failures torch.compile can fuse kernels and reduce Python overhead. But in production diffusion serving, BBuf found that its operational cost often outweighed those gains. 🔹 Slow and unpredictable Compilation may take several minutes, and complex models can take more than half an hour. Performance also changes unpredictably across compiler modes, models, and GPU generations. An optimization that works on one GPU may become a regression on another. 🔹 Hostile to manual optimization A custom fused kernel may accelerate eager execution but slow down the compiled model. Graph breaks or broader compiler fusion can erase the improvement. Developers then have to debug both their kernels and the compiler’s changing graph decisions. 🔹 Opaque and difficult to maintain Compiled traces are hard to map back to Python calls. Generated kernels are also difficult to inspect, benchmark independently, or modify. Over time, engineers are pulled into compiler-specific fusion rules and workarounds instead of optimizing the actual model. 2️⃣ The current benchmark changed the calculation In SGLang Diffusion’s latest H100 and H200 tests, torch.compile was usually slower than eager execution or roughly tied with it. Compile jobs taking longer than three minutes were stopped. Several models also encountered timeouts or runtime failures. Some models had previously benefited from compilation. But after explicit kernel optimization, most of those gaps disappeared. For models dominated by launch overhead, eager execution plus Breakable CUDA Graphs could match or beat the compiled path. 3️⃣ An Agent replaced the hidden optimizations The team did not build an elaborate multi-agent system. They used Fable with a simple optimization flow and two Kernel Design Agent skills. The Agent profiled the VAE and model components, identified bottlenecks, implemented explicit optimizations, and repeatedly checked correctness and performance. Within two weeks, this workflow optimized more than 30 diffusion models. Their eager paths reached or surpassed the performance previously provided by torch.compile. The crucial difference is ownership: these optimizations now exist as visible, testable, and maintainable code. 4️⃣ Breakable CUDA Graph replaced the other benefit Kernel fusion is only one reason to use torch.compile. The other is reducing framework overhead through graph execution. SGLang previously relied on compiler-assisted Piecewise CUDA Graphs. After dealing with substantial compiler complexity, it developed Breakable CUDA Graph, which captures reusable execution paths while allowing controlled graph breaks. SGLang Diffusion now uses the same approach. It is easier to integrate into real serving systems and can outperform the compiled path on overhead-heavy models. 5️⃣ The deeper shift is from compiler magic to Agent-owned code A general compiler applies hidden transformations at runtime. An Agent can instead inspect the workload, reproduce useful transformations as source code, and validate them on the target hardware. This makes performance work easier to profile, review, debug, and adapt. BBuf’s claim is specifically about diffusion serving, not every PyTorch workload. But within that scope, the conclusion is blunt: If Agents can reproduce the useful kernel optimizations, while CUDA Graph techniques remove framework overhead, torch.compile no longer has a unique role. 🔗 Full Reading: 🔗 Performance results: 🔗 Kernel Design Agents: 🔗 SGLang: 🔗 Agent optimization skills: #SGLang# #PyTorch# #CUDA# #AIInfra# #DiffusionModels# #AIAgents#

@ornith_ Ornith-1.5-9B python -m sglang.launch_server \ --model-path ornith-ai/Ornith-1.5-9B \ --served-model-name Ornith-1.5-9B \ --host 0.0.0.0 --port 8000 \ --context-length 262144 \ --mem-fraction-static 0.85 \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys @lmsysorg and RadixArk @radixark, and taught by Richard Chen @richardczl, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

🚀 New blog: Scaling JEV-like decision models with SGLang Decision models need a score, not prose. Classification, ranking, and agent action selection all ask the same thing: which option wins? Open-Jev, for example, scores each candidate separately with a Yes/No prompt. Serving this well raises two issues: Generate + top-k logprobs can drop the label you need, and the shared context can be recomputed for every candidate. SGLang addresses both: - /v1/score returns scores for the exact labels you request (Yes/No, A/B/C) - Multi-item scoring (MIS) computes the shared context once and keeps each candidate isolated - MIS latency stays nearly flat from 2 to 16 candidates, with 16-candidate p95 on Qwen3-8B dropping from 54.1 ms (Generate) to 20.6 ms (MIS) - MIS p95 stays under ~100 ms as load rises on Qwen3-0.6B, vs. seconds for Generate and SIS Huge thanks to the @LinkedIn team for contributing! Benchmarks and launch commands in the blog 👇

Build on K2 Horizon. •⁠ ⁠Self-host all six models with vLLM or SGLang, weights on Hugging Face •⁠ ⁠Run locally with Ollama •⁠ ⁠Use it in OpenCode and OpenClaw •⁠ ⁠Deploy through our API platform Details in IFM developer doc: Download weights: Access API Keys:

🚀 Over one trillion tokens per day, powered by Mooncake and SGLang. @ApproachingAI shares the system architecture and engineering lessons behind its token factory, running leading trillion-parameter models with SGLang HiCache + Mooncake Store. A deep dive into: 🏗️ Production architecture and system design trade-offs ⚡ Performance optimizations for trillion-token-scale workloads 🛡️ Failure isolation, fault tolerance, and recovery Real production lessons on scaling KV cache reuse while meeting strict inference SLOs. Read more:

Join @baseten, @nvidia Dynamo, and SGLang on 9/10 for an evening dedicated to the infrastructure behind reinforcement learning post-training! We will present the latest on SGLang & Miles. Miles is an RL training framework using SGLang as the rollout inference engine to make RL post-training fast and reliable. Come meet us in person! 🕐 Thursday, September 10, 6:00–9:00 PM 👉 Register now