Silicon Atlas(@Silicon_Atlas):Checking five draft tokens can take about as long as generating one, when memory bandwidth is the bottleneck. At small batch sizes, decode is often memory-bandwidth-bound. Moving the weights takes most of the time. A cheap drafter proposes a short block. The target scores the whole block in one pass, so the weights it loads serve several positions instead of one. The target accepts drafted tokens from the front and stops at the first one it rejects. The same pass also gives one more token: a fix at that position, or a bonus if the whole block held. The accept rule keeps the target's output distribution, so quality does not change. How many tokens one pass gives depends on how often the draft is right. So the drafter has to be small and still right most of the time. The discount has a limit. With enough users in one batch, the GPU becomes compute-bound and the extra verify work stops being free. So DSpark, DeepSeek's design, verifies a shorter prefix under load. DeepSeek measured it under live V4 traffic, against its earlier one-token draft (MTP-1) at the same total throughput: about 60–85% faster per-user generation on V4-Flash, 57–78% on V4-Pro. That is their stack, not a number that carries over. MLA and DSA made each target step cheaper. Speculative needs fewer of them. 3/3. Builds on MLA + DSA. Related: Expert Parallelism.

2026.10.02 15:00

Checking five draft tokens can take about as long as generating one, when memory bandwidth is the bottleneck. At small batch sizes, decode is often memory-bandwidth-bound. Moving the weights takes most of the time. A cheap drafter proposes a short block. The target scores the whole block in one pass, so the weights it loads serve several positions instead of one. The target accepts drafted tokens from the front and stops at the first one it rejects. The same pass also gives one more token: a fix at that position, or a bonus if the whole block held. The accept rule keeps the target's output distribution, so quality does not change. How many tokens one pass gives depends on how often the draft is right. So the drafter has to be small and still right most of the time. The discount has a limit. With enough users in one batch, the GPU becomes compute-bound and the extra verify work stops being free. So DSpark, DeepSeek's design, verifies a shorter prefix under load. DeepSeek measured it under live V4 traffic, against its earlier one-token draft (MTP-1) at the same total throughput: about 60–85% faster per-user generation on V4-Flash, 57–78% on V4-Pro. That is their stack, not a number that carries over. MLA and DSA made each target step cheaper. Speculative needs fewer of them. 3/3. Builds on MLA + DSA. Related: Expert Parallelism.

Silicon Atlas@Silicon_Atlas

2026.09.25 06:38

Since 2017, attention has been redesigned again and again to make decode cheaper. The changes pull on two different levers. Store less per past token: MHA (2017): every head keeps its own K and V. MQA (2019): all heads share one K and V. GQA (2023): groups of heads share one. Llama 3 uses this. MLA (2024): one short learned code per token. Shared weights let the heads use it without rebuilding past K and V. DeepSeek V2 and V3. Read fewer past tokens: Sparse attention (2025): a small index scans the past, and the main attention reads only the tokens it picks. DeepSeek V3.2, built on top of MLA. KV reads have often constrained decoding. These designs trade storage, memory traffic, and computation in different ways. MLA shows that trade clearly. The card shows the cache shrinking 57x per layer against same-size MHA, while the math per byte goes up. By my rough math, that puts MLA decode on an H800 close to where compute, not memory, becomes the limit. A third lever is fewer sequential generation steps: speculative decoding, multi-token prediction, and, for tasks that need a decision rather than prose, models like Jev that return a typed answer with probabilities instead of writing text.

Forward to community