Day-0 support for @MiniMax_AI M3.1 Flash is live in TokenSpeed. DSpark speculative decoding and MXFP8 + NVFP4 mixed precision on Blackwell. TokenSpeed is featured in the official deployment guide. 📖 👉

TokenSpeed Kernel at today’s Triton Conference poster session! Portable APIs and high-performance kernels for multi-silicon LLM inference. The AMD backend is fully rewritten in Triton + Gluon for ease of use and speed. 👇 ⭐️

685 output tokens/s per user on @Alibaba_Qwen Qwen3.8-Flash-Next. Running TokenSpeed on 4× NVIDIA GB300 with NVFP4. A look at the kernel and serving optimizations behind the results: 👉 ⭐

SMG commits are up 10x lately, and the team is the same size. That's not a typo. We've cracked how a handful of people can ship like a team 10x larger. The next step is 100x. Stay tuned. 👇

Kimi K3 on AMD Instinct MI355X, powered by TokenSpeed ⚡ 216 output tok/s per user on 8× MI355X: 1.4× the decode speed of ATOM at C1, while running higher precision. Measured on a real agentic workload: multi-turn SWE-smith, 50K+ context, 10–15 turns. Technical deep dive with @AIatAMD 🧵

TokenSpeed × Kimi K3, Part I: TP8 optimization on GB300. Now stable 3+ weeks in production across ~1,000 B300 and ~2,000 H200 GPUs. Thanks @NVIDIAAI for engineering support and GB300 NVL72 access, and community teams for their support and contributions. 👇

Releasing the @Kimi_Moonshot K3 Draft Collection — 3 draft models (EAGLE-3, DFlash2, DSpark) trained with TorchSpec and @vllm_project on @NVIDIAAI GB200. 🚀 We also shared the data recipes. More on the blog →

Adoption ≠ Contribution: Some Thoughts on Open-Source Inference Engines A few thoughts on how open-source LLM inference engines get adopted. Users usually fall into a few groups: model providers, neo labs, inference clouds, and neo clouds. There’s some overlap between them. For inference clouds and neo clouds, adoption is usually about getting the engine running. Fixes and optimizations often stay in private forks because there isn’t much incentive to push them upstream. For model providers and neo labs, moving fast is usually more important. Some keep developing on an old internal fork and gradually drift further away from upstream. Others rebase on main only occasionally. Either way, their internal fixes and optimizations don’t necessarily make it back to the open-source project. That’s why a lot of the contributions to open-source inference engines still come from the engine owner teams and vendors like NVIDIA and AMD. Also, “we run it at scale” doesn’t always tell you much about the upstream project. The internal fork can be quite different from what everyone else is using. Without a real feedback loop from production back to upstream, adoption alone doesn’t help the ecosystem that much. The adoption we value most is when people actually co-evolve with the project: run it in production, report what breaks, sync with upstream, and contribute fixes back. This is also why the engine itself is hard to turn into a product, and even harder to turn into a business. Whether you sell GPU hours, tokens, or RL services, you’re ultimately selling a service built on the same GPUs. Adding a service layer can improve the margin, but today that layer is still pretty thin, and many of the capabilities are relatively easy to replicate. The interesting part is what happens when that service layer gets deeper—better products, better user experience, and capabilities that are actually hard to reproduce. That’s where the economics start to make more sense.

OpenAI has been a pioneer in the AI industry and a strong advocate for responsible and compliant use of AI models. We’re also grateful for OpenAI’s early GPU support, which helped get TokenSpeed off the ground.

GLM 5.3 keeps the same architecture as 5.2 — no changes there, just a post-training weight update. But the jump in capability is huge. Truly impressive. TokenSpeed had day 0 support for GLM 5.3. Big thanks to the GLM team for open-sourcing such a strong model 👇

TokenSpeed is nothing without its people. We’re fortunate to have such great people, which is why we can build great things in just a few months. Keep building! 🤗

🎉 Congrats to @Zai_org on GLM-5.3-Flash — the first natively multimodal model in the GLM-5 series, and their first model combining sparse and linear attention. GLM-5.3-Flash introduces a new combination of linear attention for local dependencies and sparse attention for global context, with IndexPool keeping the indexing overhead low even at 1M-token context. This new architecture also brings a new set of inference challenges. TokenSpeed already has day-0 support for GLM-5.3-Flash, with the stack validated on both @NVIDIA and @AMD GPUs. 👇

TokenSpeed Day 0 Support for @Alibaba_Qwen 3.8 Flash Next. 🔹GDN + Qwen Sparse Attention hybrid architecture 🔹Gated residual connections 🔹N-gram embedding For N-gram embedding, TokenSpeed also supports FP8 precision. As a preview for Qwen 4, we will continue optimizing beyond Day 0.

We’re proud to be the Day 0 open-source inference engine partner for @Alibaba_Qwen 3.8. To serve this 2.4T-parameter model across multi-node @NVIDIAAI Blackwell inference, we optimized DP/EP scaling across nodes, delivering 30%+ faster performance than TP16, plus DSpark speculative decoding with single CUDA graph optimization👇

Thrilled that TokenSpeed is now part of the @PyTorch Ecosystem! 🚀 We're building an open-source LLM inference engine with a modern architecture, a next-generation scheduler, and flexible kernel abstractions for a multi-silicon future. This is just the beginning. 👇

Kimi K3 is now supported in TokenSpeed. We worked closely with the @Kimi_Moonshot team to bring Day 0 support to both NVIDIA Blackwell ((G)B200/(G)B300) and AMD Instinct (MI350X/MI355X) in just one week. ✅ Prefix caching ✅ Speculative decoding ✅ Disaggregated serving ✅ CUDA Graph decode We also share how we optimized KDA, Gated MLA, Stable LatentMoE, AttnRes, and our unified Flat KV architecture. Blog ↓

Excited to see TokenSpeed Kernel featured by @PyTorch. Portable kernel APIs enable one runtime across multiple GPU architectures without performance regrets, while keeping runtime logic and backend kernels cleanly separated. Thanks to the PyTorch for highlighting our work.