⚙ SGLang Diffusion Is Retiring torch.compile, With Help From an AI Agent After testing 35 diffusion model presets on H100 and H200 GPUs, SGLang Diffusion found that eager execution now usually matches or outperforms torch.compile. Zhihu contributor BBuf argues that this is more than a benchmark result. Agent-written optimizations and Breakable CUDA Graphs can now replace the two main benefits that torch.compile once provided. 1️⃣ The “seven sins” are three production failures torch.compile can fuse kernels and reduce Python overhead. But in production diffusion serving, BBuf found that its operational cost often outweighed those gains. 🔹 Slow and unpredictable Compilation may take several minutes, and complex models can take more than half an hour. Performance also changes unpredictably across compiler modes, models, and GPU generations. An optimization that works on one GPU may become a regression on another. 🔹 Hostile to manual optimization A custom fused kernel may accelerate eager execution but slow down the compiled model. Graph breaks or broader compiler fusion can erase the improvement. Developers then have to debug both their kernels and the compiler’s changing graph decisions. 🔹 Opaque and difficult to maintain Compiled traces are hard to map back to Python calls. Generated kernels are also difficult to inspect, benchmark independently, or modify. Over time, engineers are pulled into compiler-specific fusion rules and workarounds instead of optimizing the actual model. 2️⃣ The current benchmark changed the calculation In SGLang Diffusion’s latest H100 and H200 tests, torch.compile was usually slower than eager execution or roughly tied with it. Compile jobs taking longer than three minutes were stopped. Several models also encountered timeouts or runtime failures. Some models had previously benefited from compilation. But after explicit kernel optimization, most of those gaps disappeared. For models dominated by launch overhead, eager execution plus Breakable CUDA Graphs could match or beat the compiled path. 3️⃣ An Agent replaced the hidden optimizations The team did not build an elaborate multi-agent system. They used Fable with a simple optimization flow and two Kernel Design Agent skills. The Agent profiled the VAE and model components, identified bottlenecks, implemented explicit optimizations, and repeatedly checked correctness and performance. Within two weeks, this workflow optimized more than 30 diffusion models. Their eager paths reached or surpassed the performance previously provided by torch.compile. The crucial difference is ownership: these optimizations now exist as visible, testable, and maintainable code. 4️⃣ Breakable CUDA Graph replaced the other benefit Kernel fusion is only one reason to use torch.compile. The other is reducing framework overhead through graph execution. SGLang previously relied on compiler-assisted Piecewise CUDA Graphs. After dealing with substantial compiler complexity, it developed Breakable CUDA Graph, which captures reusable execution paths while allowing controlled graph breaks. SGLang Diffusion now uses the same approach. It is easier to integrate into real serving systems and can outperform the compiled path on overhead-heavy models. 5️⃣ The deeper shift is from compiler magic to Agent-owned code A general compiler applies hidden transformations at runtime. An Agent can instead inspect the workload, reproduce useful transformations as source code, and validate them on the target hardware. This makes performance work easier to profile, review, debug, and adapt. BBuf’s claim is specifically about diffusion serving, not every PyTorch workload. But within that scope, the conclusion is blunt: If Agents can reproduce the useful kernel optimizations, while CUDA Graph techniques remove framework overhead, torch.compile no longer has a unique role. 🔗 Full Reading: 🔗 Performance results: 🔗 Kernel Design Agents: 🔗 SGLang: 🔗 Agent optimization skills: #SGLang# #PyTorch# #CUDA# #AIInfra# #DiffusionModels# #AIAgents#

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys @lmsysorg and RadixArk @radixark, and taught by Richard Chen @richardczl, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

Built on PyTorch, Ray, SGLang, and NVIDIA Megatron-LM, Miles is an open source framework from RadixArk for large-scale LLM reinforcement learning post-training. Miles uses PyTorch for models, numerics, profiling, and extensibility; Ray for orchestration; SGLang for rollout generation; and Megatron-LM for distributed training. The framework supports asynchronous rollout and training, NCCL/RDMA weight synchronization, MoE-aware rollout/training alignment, low-precision recipes, LoRA, fault tolerance, observability, and extension points for custom algorithms and model architectures. 🔗 Read more in our latest blog from the Miles Team:

We’re excited to welcome Mooncake to the PyTorch Ecosystem! Mooncake is designed to solve the “memory wall” in LLM serving. By integrating Mooncake’s high performance KVCache transfer and storage capabilities with PyTorch native inference engines like SGLang, vLLM, and TensorRT-LLM, it unlocks new levels of throughput and scalability for large language model deployments. Mooncake enables prefill decode disaggregation, global KVCache reuse, elastic expert parallelism, and serves as a fault tolerant PyTorch distributed backend. 🔗 #PyTorch# #OpenSourceAI# #LLM# #AIInfrastructure#

There's something missing from the open vs. closed models debate that has been bothering me. The better analogy, to me, is managed vs. self-hosted infrastructure for AI. Maybe a company wants to use an open weight model because they want to do additional training on top, bringing their data and domain knowledge. This requires software and services to do additional training (e.g. Tinker and friends) as well as to serve the model (e.g. SGLang). Not all of this software is open source today! Once you have successfully trained a model with your enterprise data and expertise, you now need to deploy and serve it for customers. You can partner with an inference company to run the software and hardware. But if ownership was your primary concern, you still want to control the hardware and storage, and you now also need to run infra and secure GPU capacity. There are other valid reasons to be open. In particular, the entire industry benefits when companies training models release data or research about their work. It also allows capitalism and free markets to do their thing, increasing competition and ultimately providing better options for customers. So we should all encourage openness. The reason I prefer the managed vs. self-hosted infra framing is that we can learn from the past decade of cloud infrastructure. It's important and healthy to have both, and a great self-hosted alternative ultimately pushes the managed versions to innovate. The decision to run infra then comes down to more standard business reasons: attracting talent, the cost and maintenance of the hardware, and the importance of uptime and reliability to the business. Many businesses will say, actually, I don't want to staff and run a training and inference team, and I'm happy to pay API pricing for intelligence. And others will do the opposite and invest heavily here. We need both! As an aside, the capability of open models will reach a point where we need to be very intentional about how they are deployed. But I think this problem is solvable, whether it is sharing research early and weights later, or also open sourcing the safety stack to properly serve the model. I don't have a perfect answer here but I think the ecosystem should figure it out together. Full disclaimer, I work at a company which has released both open and closed models. There are probably people more knowledgable than myself of the open weights ecosystem. If that's you, curious if you disagree with any of this.

New open-source SOTA on agentic coding! 🚀 Ornith-1.0-397B achieves 82.4 on SWE-bench Verified  and 77.5 on Terminal-Bench 2.1, topping every open model in its class and beating Claude Opus 4.7 on both. 🤖 📦 Four sizes (9B to 397B-MoE), post-trained on Gemma 4 / Qwen 3.5, MIT licensed and globally accessible. ✨ Notably, Ornith uses RL to generate not just solution rollouts but also the scaffold that drives them. By jointly optimizing both, the model discovers better search trajectories and produces higher-quality solutions. ⚙️ Deployable on a single 8×80GB node, with vLLM and SGLang recipes in the model card.

Unlimited-OCR is now on ModelScope! A 3.3B multilingual OCR model for one-shot parsing across single images, multi-page documents, and PDFs. License: MIT🚀 🤖 📄 Full-document parsing instead of cropped-region OCR 📏 32K output length for long OCR sequences 🧩 Base and gundam image modes for different document layouts ⚙️ Transformers inference + SGLang serving with OpenAI-compatible streaming requests Built to push DeepSeek-OCR-style document parsing further.

An OpenAI friend told me he burns 300M GPT-5.5 tokens/day. The top one in his team burns billions of tokens/day. Codex coding for them every night. Databricks also gives engineers unlimited tokens. We're looking for cracked inference engineers to join us at Databricks AI to produce trillions of tokens, insanely fast. DM me if you have: - Contributed to open-source ML systems like SGLang/vLLM/PyTorch - Experience serving LLMs at large scale Databricks AI runs like a startup. Lots of exciting things to build!