Your cache is everything when it comes to inference, but how do you make sure you're keeping it around with local models? Typically when you start a server, it's a fresh slate. Then the KV cache grows, evolves, and your cache hits keep growing. But then you shut down the server, swap to a new fancy toy (model), and your cache is destroyed. Nothing we can do there, since caches are model-specific. But let's say then you want to go *back* to deploying the original model. Now *it's* KV cache was also destroyed! @sgl_project supports directly setting up hicache through some configuration arguments. The important part: * L1: GPU KV cache * L2: RAM KV Cache (usually larger, but in the world of local LLM's sometimes not!) * L3: Disk/storage-based KV Cache If you have the storage, keeping/enabling L3 allows for you to plug and play as many models as you'd like, and it'll still keep the KV Cache of your old deployment! Now, on a reboot say you have a hit on L3. Because you got a hit, it'll then keep that "warm" in L2 and L1-based caching so you don't face that terrible TTFT again. In my own learnings, you have --hicache-storage-prefetch-policy. This tells SGLang how to check and when to check if something exists in hicache in say L3. The two options I looked at were: * wait_complete * best_effort `best_effort` will check if L3 has that cached prefix, start retrieving it, but don’t block inference waiting for the full cache to arrive, reusing what’s ready and recompute anything that isn’t. Or: it'll promote whatever it can retrieve from L3 to L2/1 without blocking the request. `wait_complete` means that if L3 has that cached prefix, finish retrieving it rather than racing storage retrieval against recomputation. (These are Codex definitions) Translation: `best_effort` could have a better TTFT (probably) since it checks if we have it and then recomputes, while `wait_complete` will just fully reuse the cache and you'll wait a bit longer since it's going from cold -> hot. Don't quote me here, just where my intuition. There's a third one called `timeout`, which acts as a middle ground, still need to try it out. Last few little important parts: * SGLang will store the L3 wherever SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR is set as the env variable. * When using L3, it's based on the exact configuration and script you're currently serving with (usually with model/config information & topology, as well as tokenizer and such). I haven't tested if changing a deployment let's you reuse the L3, but my hunch is not without being careful * (Just a hunch, again still needs testing) You should probably use m.2 drives here and not HDD's if doing `wait_complete`, and possibly `best_effort` for HDDs. Need to toy with this more. But since `best_effort` recomputes, we likely can just grab it off the HDD and not worry about the transient speed.

🚀 More reliable agents with DeepSeek V4.1! XGrammar brings strict tool calling to SGLang & vLLM through Structural Tags, enforcing tool argument schemas in DeepSeek’s native format. See how the Structural Tag works 👇 Check out XGrammar 👇

Halo is not limited to supervised fine-tuning. It supports async RL and training with external environments. We are excited to partner with the @sgl_project team to make it the primary engine for rollouts.

🚀 Introducing Intern-S2-397B, our new multimodal foundation model designed for long-horizon scientific research and powerful scientific agents. 1⃣Strong general capabilities across knowledge, coding, and agents, ranking among the top open-source models. 2⃣Outstanding performance on scientific tasks including Biology-Instructions and Mol-Instructions, with strong capabilities in scientific understanding, generation, and design. 3⃣Achieves leading results among open-source models on IMO-Proof and AdvancedMathBench, reaching the level of top closed-source models such as Gemini 3.1 Pro. 4⃣Explores a new architecture for continuously expanding specialized scientific knowledge, building a stronger foundation for long-cycle scientific research. 👏Now supported by vLLM (@vllm_project ) and SGLang ( @lmsysorg ) — with more ecosystem integrations on the way. 🤗 Model: @huggingface @ModelScope2022 🤗 Try it now:

Introducing SWE-2, our closest model yet to the frontier. On leading evals, it scores on par with recent frontier models – at up to 70% lower cost. We scaled RL to multiple trillions of parameters, with a refined recipe that pushes the Pareto curve on both capabilities & cost.

🌐 Supporting open source. Expanding deployment options. We’ll work closely with the open-source community on V4.1-Flash inference support and explore more deployment options. Planning a large-scale deployment with 2,000 GPUs + a storage cluster? Let’s talk. 🔹 Model: 🔹 Paper: 6/6

Shoutout to the @sgl_project community for catching an issue in our chat template👂 We've optimized the tool-result reordering logic (early exits instead of scanning every block) and updated the templates for GLM-5.3 and GLM-5.3-Flash. Fix is live — thanks for making it better. 💪

The real world is multimodal. For AI to understand and recreate it, models need to learn across modalities. In our latest blog, we show how Miles supports that learning with a shared post-training design for VLMs and diffusion models. Link in the comments. 🔗

We’re open-sourcing Ling-3.0-flash-Fin, a finance-enhanced model for real-world workflows, and FinFIRST, an expert-built benchmark for financial search agents. Two open releases, one goal: making financial AI more accessible and verifiable.

GLM-5.3-Flash is live on Fireworks on day… 2 Why? Because we take quality very seriously. We found a benchmark discrepancy we couldn’t explain, so we delayed the launch to investigate. Day 0 (Wed): we saw 2x longer thinking on reasoning-heavy benchmarks (AIME & GPQA) for open source engines compared with @Zai_org API. Same scores, worse token efficiency. Agentic benchmarks looked good. We decided to investigate further, as overthinking might become a quality problem if max_tokens are reached Day 1 (Thu): as other non-official providers launched, their APIs had thinking in the range of open-source engines: longer than We launched a private preview endpoint with disclaimers to a few customers and worked with them to assess quality Day 2 (Fri): the official API updates. We rerun benchmarks: reasoning is now similarly long, consistent with vllm/sglang. Rest of the benchmarks, both public and internal, check out too. We launched GLM-5.3-Flash publicly: More details below

Visualized tutorials for Kimi K3 model architecture and SGLang optimizations. It is finally out now. 🫵 We spent a month polishing this blog to introduce everything you need to know about K3, and how SGLang serves it. - What's K3 secret sauce to support 1M context window on 2.8T model? - And how SGL support K3's hybrid attention architect? Link below 👇 check it out!