🔁 A way to train "think longer, get smarter" models without the gradients exploding. Title: Thinking with Looped Flows URL: Looped models that recurrently update hidden states to "think" have long struggled with unstable BPTT (backprop through time). Looped Flows fixes this by borrowing training principles from diffusion models. Here are 3 highlights. 🧠 Training recurrence without BPTT By training each step with a local loss at gradually decreasing noise levels, the model learns to keep "thinking" stably, without vanishing or exploding gradients. ⏱ More compute at inference, for free Just using a finer time grid at test time boosts accuracy, from 74.5% at 8 steps to 97.9% at 128 steps on Sudoku, with no retraining needed. 🏆 Beats prior looped models on ARC-AGI It reaches 58.8% on ARC-AGI-1 and 12.2% on ARC-AGI-2, outperforming previous looped-model approaches on both. A neat new take on test-time compute scaling: thinking longer at inference genuinely pays off. #LLMInference# #MachineLearning#

💾 TL;DR: reuse a KV cache computed on another node's GPU straight through CXL memory, up to 36.6x faster. Seagate proposes a Kubernetes-native way to share CXL memory across the cluster. Title: Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving URL: 📌 Highlights 🧩 A Kubernetes DRA driver turns CXL regions into schedulable cluster resources 🗂 No Redis/etcd needed, KV slots are tracked inside the region itself 🚀 Cross-node reuse hits 36.6x speedup over recompute at 32K-token prefixes 🎯 Hit rates reach 95.4-99.5% ⚖️ Cross-node vs same-node latency differs by only 1-4% 🐢 Still 1.15-1.70x slower than an on-GPU VRAM prefix cache hit 🔧 Measured 5.8 GB/s out of a 27 GB/s fabric ceiling, room to grow with zero-copy DMA For serving that needs to reuse long prefixes across GPU nodes, this feels like a genuinely practical option. #LLMInference# #Kubernetes#

TileRT is pushing NVIDIA GPUs into a new regime of ultra-high-interactivity inference, with a specialized decode engine delivering hundreds of tokens per second per user. In its disaggregated prefill/decode architecture, Mooncake Transfer Engine provides the high-performance data path for moving KV cache from vLLM prefill nodes to TileRT decode nodes. Read the full deep dive from SemiAnalysis: #LLMInference# #TileRT# #vLLM# #Mooncake#

Round-robin load balancing built for chat can't keep up with agent inference anymore. Title: Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs URL: 📌 Overview llm-d, an open-source distributed inference stack, hits 1.0. It turns vLLM and SGLang servers on Kubernetes into one serving pool. It is a CNCF Sandbox project with 500+ contributors. ❓ The problem Agents share long prefixes and arrive in bursts, causing redundant computation, memory pressure, and interference with interactive users. 🛠️ Key mechanisms ・Cache-aware routing with priority-based flow control ・KV cache offload to host memory, SSD, and Managed Lustre ・Disaggregated prefill/decode with safe rolling updates via DisaggregatedSet ・Time-slicing that fills idle RL capacity with async inference 📊 Results (vendor-reported) ・Agentic coding on TPU: median TTFT down 85%+, throughput 2.4x ・Cache hit rate from 35% to 70% ・About 60% fewer GPU-hours on A3 Mega ・RL duty cycle from ~40% to 70%+ 🌍 Use cases Snap, ZenteiQ, and Mistral use it. You can start by putting prefix-aware routing in front of an existing vLLM deployment. #llmd# #LLMInference#

For long-context LLM inference, should the KV cache be offloaded to disk or just recomputed on the GPU? There's no universal right answer, and this paper builds a system that decides quantitatively. Title: Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs URL: 📝 Overview The paper proposes py-kvcache, an external KV cache system for vLLM. Using io_uring for async I/O, even a Python implementation pulls near-full SSD read bandwidth of 13.5GB/s. ❗ Problem it solves Existing external KV caches like LMCache have no criteria for when to actually use them — they load from disk unconditionally even when a short prefix or a fast GPU makes recomputation cheaper. ⚙️ Methodology It introduces "scheduler-aware preloading," which starts disk reads while a request is still waiting to be scheduled, plus a "break-even gate" that rejects a load whenever it wouldn't improve time-to-first-token. 📊 Results On LongBench multi-document QA it's 6.02-7.43x faster than GPU recomputation and 2.77-3.64x faster than LMCache. On multi-turn SCBench, native vLLM read 3.4TB from disk with completion times over 1200s, while py-kvcache kept disk reads to 85GB and completion time to 480s. 🖥️ Use cases On a high-end H100, requests often don't even clear the break-even point, so skipping external caching is fine — but on a lower-end RTX 4000 Ada, external caching clearly wins, giving concrete hardware-specific guidance. #LLMInference# #vLLM#

⚡ DSpark vs DFlash: Up to 2.55× Throughput in a vLLM Test With DSpark checkpoints and vLLM support now available, parallel speculative decoding is becoming a practical serving option rather than just a research idea. Zhihu contributor kaiyuan explains how DFlash and DSpark work, then benchmarks both on the same Qwen3-4B target model. The result: DSpark reached 2.45–2.55× baseline throughput, while DFlash achieved 1.96–2.09×. 1️⃣ Why LLM Decoding Is Naturally Slow Autoregressive models generate text one token at a time. Token n+1 cannot be produced before token n. This sequential dependency limits how much parallel GPU compute can be used during decoding. Non-autoregressive generation works differently. It treats generation more like filling multiple blanks and can predict several positions in one forward pass. Parallel speculative decoding combines both ideas: 🔹 A lightweight draft model proposes several tokens in parallel. 🔹 The target model verifies them in one batch. 🔹 Accepted tokens are kept; generation restarts from the first rejection. The challenge is making the draft both fast and accurate enough to be useful. 2️⃣ Why Traditional Draft Models Hit a Wall Conventional speculative decoding often uses a smaller autoregressive model such as EAGLE. It produces higher-quality drafts, but still generates candidate tokens sequentially. Longer drafts require more draft-model forward passes. Making the draft model larger improves accuracy but also increases latency. Parallel drafters solve the latency problem by proposing an entire token block at once. However, later tokens in that block do not fully depend on earlier predictions. Their accuracy often drops quickly, creating suffix acceptance decay. 3️⃣ How DFlash Improves Parallel Drafting DFlash uses a parallel, fill-in-the-blanks-style draft model. To improve draft quality, it extracts hidden states from several layers of the target model and fuses them into an additional context representation. Each draft layer then attends to two sources: 🔹 Context derived from the target model 🔹 Representations from the draft tokens themselves The target model’s hidden states make the lightweight drafter more informed without requiring multiple autoregressive passes. DFlash can therefore propose a long block in one forward pass. But longer blocks still create a problem: the suffix is more likely to be rejected, while the target model must spend compute verifying it. 4️⃣ DSpark Adds Sequential Structure DSpark extends DFlash with semi-autoregressive generation. It first uses a parallel backbone to generate intermediate logits for multiple positions. A lightweight sequential module, implemented with an RNN or Markov head, then produces the draft tokens from left to right. This small sequential step restores dependencies inside the block without giving up most of the parallel speed. It also predicts a confidence value for every token: the probability that the token will survive target-model verification if the previous prefix is accepted. The result is a stronger draft with less suffix decay. 5️⃣ Verification Length Becomes Dynamic DSpark does not automatically send the entire draft to the target model. Its Hardware-Aware Prefix Scheduler considers: 🔹 The survival probability of each draft prefix 🔹 The current batch size and system load 🔹 A profiled steps-per-second curve for the hardware Verifying one more token may increase the expected accepted length. But it also enlarges the verification batch and can reduce processing speed. The scheduler expands each prefix only while estimated throughput continues improving. Low-confidence suffix tokens are discarded before they consume target-model compute. Draft long, but verify only the prefix that is still worth verifying. 6️⃣ The vLLM Test Setup The author tested both methods under the same environment: 🔹 8× NVIDIA A800-SXM4-80GB 🔹 Qwen3-4B target model 🔹 DSpark block-7 and DFlash block-16 draft models 🔹 vLLM 0.26.0 The initial comparison used DSpark with four speculative tokens and DFlash with seven. On 250 GSM8K questions: ✅ DSpark: 35.2% accuracy, 0.75s average latency ✅ DFlash: 31.6% accuracy, 0.84s average latency On 250 MMLU questions: ✅ DSpark: 28.8% accuracy, 0.24s average latency ✅ DFlash: 27.6% accuracy, 0.27s average latency DSpark reduced average latency by roughly 11% in both tests. 7️⃣ Throughput Is the Stronger Result Because the two methods used different speculative-token settings, the author swapped those parameters and tested again. 🔹 Original settings: DSpark=7, DFlash=4 DSpark reached 584 tok/s, while DFlash reached 449 tok/s. ✅ DSpark was 1.30× faster. 🔹 Swapped settings: DSpark=4, DFlash=7 DSpark reached 561 tok/s, while DFlash reached 480 tok/s. ✅ DSpark remained 1.17× faster. Changing num_speculative_tokens affected throughput by less than 7%. DSpark remained faster in both configurations, showing that its advantage did not come from receiving a more favorable draft length. Compared with the 229 tok/s baseline: 🔹 DSpark delivered 2.45–2.55× throughput. 🔹 DFlash delivered 1.96–2.09× throughput. At matched settings, DSpark stayed roughly 20% faster than DFlash. ⚠️ Do Not Overread the Accuracy Numbers The accuracy differences are less conclusive. MMLU results varied by about one percentage point. On GSM8K, even the baseline changed from 29.2% to 34.4% across two runs. The author attributes this to different vLLM batch compositions changing floating-point accumulation order. That can alter a small number of token choices even with temperature=0. So the test strongly supports a throughput advantage. It does not establish that speculative decoding improves model intelligence. 💡 The Practical Takeaway DFlash proves that parallel drafting can generate many candidates cheaply. DSpark adds the two components needed for production serving: 🔹 Lightweight sequential modeling to improve draft quality 🔹 Load-aware scheduling to avoid unnecessary verification Its real contribution is not simply drafting more tokens. It is deciding which tokens are still worth verifying under the current serving load. 🔗 DeepSpec: 🔗 Test notebook: 🔗 Full Reading: #DSpark# #DFlash# #SpeculativeDecoding# #vLLM# #LLMInference# #AIInfrastructure# #DeepSeek#

LLM inference produces one token at a time. An 8B flow language model challenges that assumption. Title: Large Flow Language Models URL: 🌊 This work adapts pretrained autoregressive models into Lafa, a blockwise flow language model. It trains on fewer than 30B tokens and reaches 2.54x the throughput of AR at 7B with comparable quality. ⚡ Highlight 1: Asynchronous flows Each token in a block gets its own time, with the denoising schedule learned from the loss. A left-to-right tendency emerges on its own, without being imposed. 🧠 Highlight 2: Self-speculative decoding A one-step flow drafts tokens, and the AR head kept inside the same model verifies them. The output matches greedy AR decoding while producing roughly 2.5 to 5 tokens per forward pass. 🪶 Highlight 3: Less training, better quality Built on Qwen2.5 and Qwen3, it uses at least 18x fewer training tokens than Dream, LLaDA, and Efficient-DLM. The 7B average accuracy is 60.9 versus 58.4 for AR SFT. A strong sign that continuous flows can be a practical foundation for parallel generation. #LanguageModels# #FastInference#

Revolut’s blended LLM inference cost is now $0.49 per million tokens. That figure covers external models, self-hosted GPU capacity (including idle time) and tool overhead. Coding assistance sits even lower, at roughly $0.30 on self-hosted GLM. This matters because AI usage hits the bottom line, and engineers don't want to be rationed. Revolut is showing that a regulated fintech can build the internal capability to run cheap, high-volume AI at scale — the kind of constraint removal that affects how fast a company ships product. It is also very on-brand: keep as much as possible in-house, especially with self-hosted open-weight models. Given the existing NVIDIA collaboration on PRAGMA and the ongoing build-out of self-hosted capacity, this looks more like a preview of where Revolut is heading than a one-off optimisation.

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers don’t mean it’s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: ↳ When you download a model, it compiles that model’s kernels locally. ↳ It tries different kernel configurations and keeps whichever runs fastest on your chip. ↳ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: (don’t forget to star it 🌟) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don’t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.

Full-day hands-on workshops on LLM inference, physical AI, and post-training, plus in-person certification. Fri Oct 16, Volkswagen Drive Forum, Berlin. Part of Nebius Build Week.