⚡ DSpark vs DFlash: Up to 2.55× Throughput in a vLLM Test With DSpark checkpoints and vLLM support now available, parallel speculative decoding is becoming a practical serving option rather than just a research idea. Zhihu contributor kaiyuan explains how DFlash and DSpark work, then benchmarks both on the same Qwen3-4B target model. The result: DSpark reached 2.45–2.55× baseline throughput, while DFlash achieved 1.96–2.09×. 1️⃣ Why LLM Decoding Is Naturally Slow Autoregressive models generate text one token at a time. Token n+1 cannot be produced before token n. This sequential dependency limits how much parallel GPU compute can be used during decoding. Non-autoregressive generation works differently. It treats generation more like filling multiple blanks and can predict several positions in one forward pass. Parallel speculative decoding combines both ideas: 🔹 A lightweight draft model proposes several tokens in parallel. 🔹 The target model verifies them in one batch. 🔹 Accepted tokens are kept; generation restarts from the first rejection. The challenge is making the draft both fast and accurate enough to be useful. 2️⃣ Why Traditional Draft Models Hit a Wall Conventional speculative decoding often uses a smaller autoregressive model such as EAGLE. It produces higher-quality drafts, but still generates candidate tokens sequentially. Longer drafts require more draft-model forward passes. Making the draft model larger improves accuracy but also increases latency. Parallel drafters solve the latency problem by proposing an entire token block at once. However, later tokens in that block do not fully depend on earlier predictions. Their accuracy often drops quickly, creating suffix acceptance decay. 3️⃣ How DFlash Improves Parallel Drafting DFlash uses a parallel, fill-in-the-blanks-style draft model. To improve draft quality, it extracts hidden states from several layers of the target model and fuses them into an additional context representation. Each draft layer then attends to two sources: 🔹 Context derived from the target model 🔹 Representations from the draft tokens themselves The target model’s hidden states make the lightweight drafter more informed without requiring multiple autoregressive passes. DFlash can therefore propose a long block in one forward pass. But longer blocks still create a problem: the suffix is more likely to be rejected, while the target model must spend compute verifying it. 4️⃣ DSpark Adds Sequential Structure DSpark extends DFlash with semi-autoregressive generation. It first uses a parallel backbone to generate intermediate logits for multiple positions. A lightweight sequential module, implemented with an RNN or Markov head, then produces the draft tokens from left to right. This small sequential step restores dependencies inside the block without giving up most of the parallel speed. It also predicts a confidence value for every token: the probability that the token will survive target-model verification if the previous prefix is accepted. The result is a stronger draft with less suffix decay. 5️⃣ Verification Length Becomes Dynamic DSpark does not automatically send the entire draft to the target model. Its Hardware-Aware Prefix Scheduler considers: 🔹 The survival probability of each draft prefix 🔹 The current batch size and system load 🔹 A profiled steps-per-second curve for the hardware Verifying one more token may increase the expected accepted length. But it also enlarges the verification batch and can reduce processing speed. The scheduler expands each prefix only while estimated throughput continues improving. Low-confidence suffix tokens are discarded before they consume target-model compute. Draft long, but verify only the prefix that is still worth verifying. 6️⃣ The vLLM Test Setup The author tested both methods under the same environment: 🔹 8× NVIDIA A800-SXM4-80GB 🔹 Qwen3-4B target model 🔹 DSpark block-7 and DFlash block-16 draft models 🔹 vLLM 0.26.0 The initial comparison used DSpark with four speculative tokens and DFlash with seven. On 250 GSM8K questions: ✅ DSpark: 35.2% accuracy, 0.75s average latency ✅ DFlash: 31.6% accuracy, 0.84s average latency On 250 MMLU questions: ✅ DSpark: 28.8% accuracy, 0.24s average latency ✅ DFlash: 27.6% accuracy, 0.27s average latency DSpark reduced average latency by roughly 11% in both tests. 7️⃣ Throughput Is the Stronger Result Because the two methods used different speculative-token settings, the author swapped those parameters and tested again. 🔹 Original settings: DSpark=7, DFlash=4 DSpark reached 584 tok/s, while DFlash reached 449 tok/s. ✅ DSpark was 1.30× faster. 🔹 Swapped settings: DSpark=4, DFlash=7 DSpark reached 561 tok/s, while DFlash reached 480 tok/s. ✅ DSpark remained 1.17× faster. Changing num_speculative_tokens affected throughput by less than 7%. DSpark remained faster in both configurations, showing that its advantage did not come from receiving a more favorable draft length. Compared with the 229 tok/s baseline: 🔹 DSpark delivered 2.45–2.55× throughput. 🔹 DFlash delivered 1.96–2.09× throughput. At matched settings, DSpark stayed roughly 20% faster than DFlash. ⚠️ Do Not Overread the Accuracy Numbers The accuracy differences are less conclusive. MMLU results varied by about one percentage point. On GSM8K, even the baseline changed from 29.2% to 34.4% across two runs. The author attributes this to different vLLM batch compositions changing floating-point accumulation order. That can alter a small number of token choices even with temperature=0. So the test strongly supports a throughput advantage. It does not establish that speculative decoding improves model intelligence. 💡 The Practical Takeaway DFlash proves that parallel drafting can generate many candidates cheaply. DSpark adds the two components needed for production serving: 🔹 Lightweight sequential modeling to improve draft quality 🔹 Load-aware scheduling to avoid unnecessary verification Its real contribution is not simply drafting more tokens. It is deciding which tokens are still worth verifying under the current serving load. 🔗 DeepSpec: 🔗 Test notebook: 🔗 Full Reading: #DSpark# #DFlash# #SpeculativeDecoding# #vLLM# #LLMInference# #AIInfrastructure# #DeepSeek#

🚀 @deepseek_ai's DSpark speculative decoding now runs natively in vLLM! What it is: a semi-autoregressive drafter that proposes several tokens in parallel with non-causal sliding-window attention, then verifies them in a single pass. Output stays identical, decoding takes fewer steps. How vLLM runs it: it reuses the existing SparseMLA backends instead of custom attention kernels, captures the full draft backbone and sampling loop in one CUDA graph, and works with prefix caching and FP8 KV cache. Performance on DeepSeek-V4-Pro-DSpark (verified on NVIDIA 8×B300 GPUs): - ~250 tokens/s at batch size 1 - average acceptance length ~5 - 12-42% higher acceptance than MTP across draft depths Run with vLLM nightly today: vllm serve deepseek-ai/DeepSeek-V4-Pro-DSpark -tp 8 --trust-remote-code --kv-cache-dtype fp8 --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}' DSpark Core PR: Thanks @deepseek_ai for open-sourcing DSpark, and to @NVIDIAAI and the vLLM community for landing it! 🙏

🚀 We’ve open-sourced AngelSpec, an end-to-end speculative decoding framework supporting both training and deployment. On Hy3-A21B, DFly delivers a 1.98–2.40× end-to-end speedup over autoregressive decoding across tested concurrency levels from 4 to 64, with 10.5–11.8% higher throughput than DFlash. Training code and Hy3-A21B MTP/DFly drafter weights are now available: GitHub: Paper: Docs: Hugging Face: ModelScope: #Hy3# #AngelSpec# #OpenSource#

~5 minutes on what's new in @vllm_project v0.24 from core maintainer @mgoin_, including Blackwell RTX SparseMLA for DeepSeek V4 and GLM 5.2, dynamic speculative decoding, streaming parser engine, LibTorch stable ABI migration, and more.

Benchmarking @NVIDIAAI's Nemotron Puzzle 75B locally on the GX10. NVFP4 via vLLM's OpenAI API, MTP speculative decoding, forced 1,500-token generations. 🏃‍♀️22.75 tok/s in a single session into 88.85 cumulative at 7 sessions. 🧍Baseline without MTP: ~16.3. Scripts + setup:

DeepSeek is the GOAT. 🐳 They just published DSpark, a new speculative decoding method that boosts throughput by 51% to 400%. They also open-sourced DeepSpec, the training framework behind it. This is the real open AI.

As an AI Engineer. Please learn: - Harness engineering, not just prompt engineering - Prompt caching vs. semantic caching tradeoffs - KV cache management at scale - Speculative decoding vs quantization - Structured output failures & fallback chains - Evals (LLM-as-judge + human evals) - Cost attribution per feature, not just per model - Agent guardrails & loop budgets - LLM observability as a first-class discipline - Model routing & graceful fallback logic - Knowing when to fine-tune vs. in-context learning

⚡ Stop treating intelligence and efficiency as separate. GPT-5.6 maximizes intelligence per token to deliver equal-or-better performance more cheaply and quickly. Title: How GPT-5.6 fuses frontier intelligence with frontier efficiency URL: ⚡ Overview GPT-5.6 is trained to optimize both task success and efficiency, taking a more direct path through tasks. OpenAI calls it their greatest intelligence-per-token efficiency yet. 🧩 Problem Solved Frontier models are smart, but reasoning tokens, latency, and cost are the wall in production. GPT-5.6 makes efficiency a first-class goal, pushing the performance-vs-cost tradeoff outward. 🛠 Methodology & Lineup ・Sol: flagship for frontier reasoning and long-horizon agentic work ・Terra: everyday balanced model, GPT-5.5-competitive at about half the cost ・Luna: fastest and cheapest (~80% less than Sol) On serving: improved speculative decoding gives 15%+ better token generation, and GPU kernel improvements cut serving cost 20%. 📊 Results On the Artificial Analysis Coding Agent Index, Sol (max reasoning) sets a new SOTA of 80, beating Fable 5 by +2.8 while using under half the output tokens, half the time, and ~1/3 less cost. On ExploitBench it matches Mythos Preview using ~1/3 of the output tokens. #GPT56# #OpenAI#

🧠 A transformer can always attend back to the past, so it has no structural incentive to compress history into a compact latent state. That means it tends to hoard information instead of summarizing it — and that hurts generalization. The fix proposed here is to add a light auxiliary objective to next-token prediction: also predict your own next latent. By learning to guess its next hidden state given the next token, the model grows a compact internal state with consistent transition rules — theoretically, a "belief state" that's a sufficient statistic for predicting the future. The transformer stays parallel while a lightweight dynamics model enforces temporal consistency — essentially co-training a transformer and an RNN in parallel. And the payoff is concrete. On a Manhattan-taxi world model the latent rank is ~3× more compact than GPT's; on reasoning and planning it looks ahead correctly instead of taking shortcuts (Countdown 54.8% vs ~39%). On a 1.3B language model it preserves quality while variable-length self-speculative decoding makes inference 3.3× faster (MTP/JTP cap out around 1.7×). Best of all, you only add a small MLP at training time — inference stays a single transformer. Next-Latent Prediction Transformers Learn Compact World Models #Transformer# #WorldModels#

We're introducing GLM-5.2, our latest flagship model for long-horizon tasks. It marks a substantial leap in long-horizon task capability over its predecessor GLM-5.1 and, for the first time, delivers that capability on a solid 1M-token context. GLM-5.2's new capabilities include: Solid 1M Context: A solid 1M-token context that stably sustains long-horizon work Advanced Coding with Flexible Effort: Stronger coding capabilities with multiple thinking effort levels to balance performance and latency Improved Architecture: We propose IndexShare, which reuses the same indexer across every four sparse attention layers, reducing per-token FLOPs by 2.9× at a 1M context length. We also improve GLM-5.2’s MTP layer for speculative decoding, increasing the acceptance length by up to 20% Pure Open: An MIT open-source license — no regional limits, technical access without borders Supporting long-horizon tasks starts with making long context engineering-usable: the model must maintain quality across long, messy coding-agent trajectories, not just accept more tokens. A 1M context is easy to claim, but much harder to keep reliable under real engineering pressure. To this end, we substantially expanded 1M-context training for coding-agent scenarios, covering large-scale implementation, automated research, performance optimization, and complex debugging. The result is a long-context system that is not only wide in scope, but solid in execution: a practical substrate for sustained engineering work. This capability is reflected in GLM-5.2's performance on three long-horizon coding benchmarks. FrontierSWE measures whether an agent can complete open-ended technical projects at the scale of hours to tens of hours, spanning systems optimization, large-scale code construction, and applied ML research. On this benchmark, GLM-5.2 trails Opus 4.8 by only 1%, while edging out GPT-5.5 by 1% and Opus 4.7 by 11%. On PostTrainBench, where each agent is given an H100 GPU and evaluated by how much it can improve small models through post-training, GLM-5.2 outperforms both Opus 4.7 and GPT-5.5, ranking second only to Opus 4.8. On SWE-Marathon, an ultra-long-horizon software engineering benchmark covering tasks such as building compilers, optimizing kernels, and developing production-grade services, GLM-5.2 still has room to grow, trailing Opus 4.8 by 13% while remaining second only to the Opus series. Across all three benchmarks, GLM-5.2 is the highest-ranked open-source model, showing that its 1M context has translated into practical long-horizon delivery capability.