⚡ DSpark vs DFlash: Up to 2.55× Throughput in a vLLM Test With DSpark checkpoints and vLLM support now available, parallel speculative decoding is becoming a practical serving option rather than just a research idea. Zhihu contributor kaiyuan explains how DFlash and DSpark work, then benchmarks both on the same Qwen3-4B target model. The result: DSpark reached 2.45–2.55× baseline throughput, while DFlash achieved 1.96–2.09×. 1️⃣ Why LLM Decoding Is Naturally Slow Autoregressive models generate text one token at a time. Token n+1 cannot be produced before token n. This sequential dependency limits how much parallel GPU compute can be used during decoding. Non-autoregressive generation works differently. It treats generation more like filling multiple blanks and can predict several positions in one forward pass. Parallel speculative decoding combines both ideas: 🔹 A lightweight draft model proposes several tokens in parallel. 🔹 The target model verifies them in one batch. 🔹 Accepted tokens are kept; generation restarts from the first rejection. The challenge is making the draft both fast and accurate enough to be useful. 2️⃣ Why Traditional Draft Models Hit a Wall Conventional speculative decoding often uses a smaller autoregressive model such as EAGLE. It produces higher-quality drafts, but still generates candidate tokens sequentially. Longer drafts require more draft-model forward passes. Making the draft model larger improves accuracy but also increases latency. Parallel drafters solve the latency problem by proposing an entire token block at once. However, later tokens in that block do not fully depend on earlier predictions. Their accuracy often drops quickly, creating suffix acceptance decay. 3️⃣ How DFlash Improves Parallel Drafting DFlash uses a parallel, fill-in-the-blanks-style draft model. To improve draft quality, it extracts hidden states from several layers of the target model and fuses them into an additional context representation. Each draft layer then attends to two sources: 🔹 Context derived from the target model 🔹 Representations from the draft tokens themselves The target model’s hidden states make the lightweight drafter more informed without requiring multiple autoregressive passes. DFlash can therefore propose a long block in one forward pass. But longer blocks still create a problem: the suffix is more likely to be rejected, while the target model must spend compute verifying it. 4️⃣ DSpark Adds Sequential Structure DSpark extends DFlash with semi-autoregressive generation. It first uses a parallel backbone to generate intermediate logits for multiple positions. A lightweight sequential module, implemented with an RNN or Markov head, then produces the draft tokens from left to right. This small sequential step restores dependencies inside the block without giving up most of the parallel speed. It also predicts a confidence value for every token: the probability that the token will survive target-model verification if the previous prefix is accepted. The result is a stronger draft with less suffix decay. 5️⃣ Verification Length Becomes Dynamic DSpark does not automatically send the entire draft to the target model. Its Hardware-Aware Prefix Scheduler considers: 🔹 The survival probability of each draft prefix 🔹 The current batch size and system load 🔹 A profiled steps-per-second curve for the hardware Verifying one more token may increase the expected accepted length. But it also enlarges the verification batch and can reduce processing speed. The scheduler expands each prefix only while estimated throughput continues improving. Low-confidence suffix tokens are discarded before they consume target-model compute. Draft long, but verify only the prefix that is still worth verifying. 6️⃣ The vLLM Test Setup The author tested both methods under the same environment: 🔹 8× NVIDIA A800-SXM4-80GB 🔹 Qwen3-4B target model 🔹 DSpark block-7 and DFlash block-16 draft models 🔹 vLLM 0.26.0 The initial comparison used DSpark with four speculative tokens and DFlash with seven. On 250 GSM8K questions: ✅ DSpark: 35.2% accuracy, 0.75s average latency ✅ DFlash: 31.6% accuracy, 0.84s average latency On 250 MMLU questions: ✅ DSpark: 28.8% accuracy, 0.24s average latency ✅ DFlash: 27.6% accuracy, 0.27s average latency DSpark reduced average latency by roughly 11% in both tests. 7️⃣ Throughput Is the Stronger Result Because the two methods used different speculative-token settings, the author swapped those parameters and tested again. 🔹 Original settings: DSpark=7, DFlash=4 DSpark reached 584 tok/s, while DFlash reached 449 tok/s. ✅ DSpark was 1.30× faster. 🔹 Swapped settings: DSpark=4, DFlash=7 DSpark reached 561 tok/s, while DFlash reached 480 tok/s. ✅ DSpark remained 1.17× faster. Changing num_speculative_tokens affected throughput by less than 7%. DSpark remained faster in both configurations, showing that its advantage did not come from receiving a more favorable draft length. Compared with the 229 tok/s baseline: 🔹 DSpark delivered 2.45–2.55× throughput. 🔹 DFlash delivered 1.96–2.09× throughput. At matched settings, DSpark stayed roughly 20% faster than DFlash. ⚠️ Do Not Overread the Accuracy Numbers The accuracy differences are less conclusive. MMLU results varied by about one percentage point. On GSM8K, even the baseline changed from 29.2% to 34.4% across two runs. The author attributes this to different vLLM batch compositions changing floating-point accumulation order. That can alter a small number of token choices even with temperature=0. So the test strongly supports a throughput advantage. It does not establish that speculative decoding improves model intelligence. 💡 The Practical Takeaway DFlash proves that parallel drafting can generate many candidates cheaply. DSpark adds the two components needed for production serving: 🔹 Lightweight sequential modeling to improve draft quality 🔹 Load-aware scheduling to avoid unnecessary verification Its real contribution is not simply drafting more tokens. It is deciding which tokens are still worth verifying under the current serving load. 🔗 DeepSpec: 🔗 Test notebook: 🔗 Full Reading: #DSpark# #DFlash# #SpeculativeDecoding# #vLLM# #LLMInference# #AIInfrastructure# #DeepSeek#

we've accumulated nearly $100,000 to give bankr ecosystem projects FREE LLM inference. if youre building on bankr you can expect to receive free inference soon. before we start distributions we need to ship something that i think the entire industry is going to enjoy. more on that soon!

# Practices for Embedding AI Agents in Software # Adaptive Timeout & Budget-Bounded Retry 🎯 The Hook Still retrying exactly 3 times? A single LLM retry can burn thousands of tokens. In the agent era, retry limits should be budgets, not counters. 🔥 The Problem Agent execution mixes operations with wildly different latency profiles: tool calls (seconds) vs. LLM inference (tens of seconds to minutes). A single fixed timeout either waits too long for tools or cuts off inference too early. Fixed-count retries ignore cost: three LLM retries can blow through your token budget, while three network retries may not be enough. And treating a 429 rate-limit the same as a schema violation wastes resources on retries that cannot succeed. 💡 The Pattern Adaptive Timeout & Budget-Bounded Retry sets timeouts per operation class (tool call, LLM inference, full session) and caps retries by remaining token budget rather than a fixed count. Errors are classified into three types: transient (429, 5xx) handled with exponential backoff, content-caused (schema violations) handled with self-correction, and context overflow handled with summarization or splitting. As budget consumption crosses thresholds, the system degrades gracefully: falling back to lighter models, returning partial results, and ultimately failing fast. ✅ When to Use Use when: - The agent combines operations with different latency characteristics - Retry cost is non-trivial (includes LLM inference) - Both transient network errors and content-caused errors can occur Don't use when: - A single LLM call with a fixed timeout is sufficient - Retries are prohibited (non-idempotent writes without idempotency keys) ⚠️ Pitfalls - Ignoring the Retry-After header on 429 responses and relying solely on your own backoff will trigger further throttling by the provider - Retrying non-idempotent writes without idempotency keys causes double execution. Either add keys or skip retries entirely for those operations - Stuffing raw error messages into self-correction prompts bloats the context window and triggers context-length errors. Summarize error feedback to roughly 200 tokens 🔧 Implementation Approach - Define timeouts per operation class: roughly 10-30s for tool calls, 60-120s total for LLM inference (with 5-15s inter-token timeout during streaming), and session-level deadlines for the full execution - Classify errors into three types and route accordingly: transient errors (429/5xx/timeout) use exponential backoff with jitter, content errors (schema violations etc.) trigger self-correction by appending an error summary to context, and context overflow triggers summarization or splitting - Cap retries by remaining token budget rather than a fixed count. Allow transient retries up to 90% of the budget and self-correction retries up to 70% - Deploy independent circuit breakers (Closed/Open/Half-Open) per provider, and on breaker open, descend a degradation ladder: lighter model, cached response, static fallback, then fail-fast - Implement a provider abstraction layer with a common interface so fallback routing is transparent to callers, and attach a degradation-level tag to response metadata #AIAgents# #SoftwareArchitecture#

AMD AI Takeaways Note •AI accelerator market now seen at $1.4T by 2030 (up from $1T prior). We also raises forecast to $1T by 2028, backed by our bottom-up analysis •Server CPU TAM also boosted to >$220B by 2030. Agentic AI expected to be ~50% of TAM, with CPU:GPU ratio tightening toward 1:1. •MI455X is a major leap: 34x higher throughput & 18x lower token cost vs MI355X. Already outperforming Nvidia B200 in some LLM inference. •MI500 brings optical interconnects, ahead of Nvidia Rubin Ultra in key areas (HBM, 4-die package, scale-up domain). •Venice CPU strongest for AI workloads: 2.2x/2.0x higher throughput vs competitors. We estimate ~300k CoWoS builds expected in 2027 → big ASP uplift. 📈 No change to estimates and TP #AMD# #AdvancingAI#

"early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art. A detailed technical report on performance will be presented in the coming months. The architecture reduces data movement and balances compute, memory, and networking resources to achieve realized utilization much closer to theoretical peak performance. Jalapeño is a blank-slate design for modern LLM inference, not a general-purpose accelerator adapted from earlier AI workloads. It is informed by the systems OpenAI runs every day across ChatGPT, Codex, the API, and future agentic products, while also being designed for current and future LLMs across the industry. The goal is to combine the power and throughput of today’s leading AI accelerators with latency closer to the fastest specialized inference systems, making Jalapeño well suited for interactive LLM products at scale. That is the full-stack advantage. OpenAI is not only developing frontier models or building products on top of them; it is designing the infrastructure underneath them: chip architecture, kernels, memory systems, networking, scheduling, deployment systems, and product experience. Because OpenAI operates across the stack, each layer can be optimized around the same goal: making its models faster, more reliable, and more affordable for users."

$CBRS lists May 14 — set to be the biggest US IPO 2026 Live on StableStock Day 1 Cerebras Systems, the most credible challenger to NVIDIA in AI compute. Wafer-scale chips, 880× more on-chip memory than GPUs, 15× faster on LLM inference. $20B+ OpenAI contract, AWS partnership, $5B revenue by 2030 Trade $CBRS on StableStock May 14 from the moment it opens on

The Underpriced Truth: Agentic AI Is a Paradigm Shift Centered on Memory 1/ The market will slowly realize: Agentic AI is memory-centric, not compute-centric. The new hardware stack is: ① Memory — HBM / DRAM / NAND ② Parallel compute — GPU / ASIC ③ Coordinator — CPU CPUs stopped doing the heavy lifting a long time ago. This isn't a cycle. It's a paradigm. 🧵👇 2/ First principles Humanity's ultimate pursuit of intelligence has always been two things: Infinite memory + infinite compute. When we say someone is smart, we mean two things: "good memory" + "fast thinking." Machine intelligence is walking the exact same path. 3/ The story the market already understands: HBM LLM inference's decode stage is a textbook memory-bound workload. Every token generated → drag the entire KV cache across memory. Bandwidth too low → expensive GPUs sit idle. That's why every new GPU generation ships with more HBM bandwidth and capacity. 4/ The story the market is missing The "1M context" you keep hearing about? It is not assembled inside the GPU inference cluster. So where is it actually built? 5/ It's built on the traditional servers running the agentic system Those CPU + huge-DRAM servers are quietly doing the heaviest lifting: • loading user long-term & short-term memory • loading the agent's system spec / prompt • loading skill / tool / subagent definitions • compressing the context once it overflows 1M tokens All of this lives in DRAM, not HBM. 6/ Compare this to the previous era In the web / mobile era, we barely stored any user context at all. Only search / recsys / ads kept a small user profile — maybe 1/20, even 1/100 of the data volume an agentic system needs today. That asymmetry is the real overlooked inflection point. 7/ The supply chain is already telling this story Server CPU : DRAM ratio is climbing fast: • Web / Mobile era: 1 core : 4 GB • Agentic AI today: 1 core : 16 GB • Deep agentic future: 1 core : 64 GB and beyond 8/ And it's NOT just "4x more memory" Under agentic workloads, a single CPU serves a fraction of the users it used to. When the entire IT stack migrates to agentic: • CPU count grows several-fold to ~10x • DRAM total grows tens-fold to ~100x That's the part nobody is pricing in. 9/ The conclusion Agentic AI is a paradigm shift centered on storage + parallel compute. The software paradigm changed. The hardware paradigm changed with it. Only those who deeply understand the technology will see it: This isn't a memory cycle. It's a memory paradigm. 10/ Time horizon Given how early we still are on: • user adoption rate • depth of usage per user We are at least 5 years away from the cyclical top of this memory wave. (Zoom out far enough and everything is a cycle — but this one is nowhere near peak.) $MU $DRAM $SNDK

We’re excited to bring #DeepSeek# V4 Pro to Bitdeer AI Model Studio on Day0 launch — expanding access to one of the latest generation of reasoning-focused models. Here are the key highlights: 🔹1M-token context window (vs ~128K in V3.2) → Enables full codebase, long-document, and multi-step workflow processing 🔹Stronger reasoning performance → Improved accuracy across complex, multi-step tasks and coding scenarios 🔹Enhanced agentic capabilities → Better tool use, task decomposition, and execution for agent workflows 🔹Built for production use cases → More suitable for coding agents, enterprise AI workflows, and large-scale reasoning applications Through Bitdeer AI Model Studio, teams can access DeepSeek V4 Pro through a simple API experience, making it easier to test, integrate, and scale advanced AI models without managing complex infrastructure. Use DeepSeek V4 Pro now⚡️: #opensource# #LLM# #inference# #neocloud#

Kimi K2.6 has landed, and it is live on Baseten! We have baked in multiple inference optimizations so that you can leverage Kimi K2.6 in production right away. To run Kimi K2.6, Baseten uses: -> The Baseten Inference Stack with advanced optimizations, including KV-aware routing -> NVFP4 weights to unlock maximum performance on NVIDIA Blackwell GPUs -> Multimodal hierarchical caching for low-latency vision input -> Prefill-decode disaggregation for LLM inference optimization. Try it now at:

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys @lmsysorg and RadixArk @radixark, and taught by Richard Chen @richardczl, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!