DeepSeek V4.1 Gives Prefill and Decode Different Compute Paths Prompt tokens mostly traverse 20 layers; generated tokens traverse all 40. This Causal Encoder-Decoder design nearly halves long-input prefill while preserving autoregressive generation. Zhihu contributor 潜龙勿用, Changxin Ke(柯昌鑫), a graduate researcher at ICT, CAS, explains how its architecture and post-training were designed together. 1️⃣ A causal encoder, not T5 The 40-layer backbone is split into a 20-layer causal encoder and a 20-layer decoder. Both remain causal. The encoder processes the prompt and supplies the decoder’s global KV. Most prompt tokens avoid the decoder stack, while generated tokens still run through all 40 layers. That means 8B active parameters per prefill token versus 16B per decode token. 2️⃣ Most layers share global memory CSA2 uses three modes: Full creates global KV and an index; Reindex shares the KV but selects new positions; Reuse shares both. Only four layers create independent global KV, four reindex it, and 30 reuse both KV and the latest index. Each layer retains its own query and local SWA state. The first Full layer builds up to 16,384 candidates. Later layers search this pool for their Top-512. From 4K to 1M context, decode FLOPs per token rise only about 25%. 3️⃣ Serving approximations enter training Exact reconstruction of decoder SWA states would replay 2,560 prompt tokens. V4.1 replays only the final 128 encoder outputs and trains the model to tolerate the approximation. With FP4 global KV, cache falls to 890 bytes per token, persistent cache to roughly one eighth of V4-Flash, and prefill compute close to half. Bounded replay and constrained retrieval are not last-minute serving tricks. The model experiences them during post-training. 4️⃣ Post-training is an evolving Agent system Each RL task combines a problem, environment, and verifier. New trajectories can reveal shortcuts, broken environments, or verifier errors and send the task back for repair. V4.1 trains across multiple harnesses, merges checkpoints between RL runs, and finishes with on-policy distillation from more than 40 teachers. Raising reasoning effort from 25 to 100 increases output length about 2.5×, while average Pass@1 across eight benchmarks rises from 67.1% to 76.3%. ✅ The real design choice V4.1 aligns model structure, cache policy, retrieval limits, training environments, and inference around long-running Agents. The model learns under the same constraints the deployed system will actually impose. 🔗 Full analysis: #DeepSeek# #DeepSeekV41# #LLMArchitecture# #AIAgents# #LongContext# #AIInfra#

DeepSeek V4.1 Flash Is an Architecture Reset, Not Just a Cheaper Model Despite having more total and active parameters than its predecessor, V4.1 Flash cuts working KV cache to one quarter and persistent cache storage to one eighth. Zhihu contributor Exhalation explains how it does this by deleting old modules, sharing KV states, and recomputing local context. 1️⃣ DeepSeek removed its own previous ideas 🔹 MTP: external draft models such as DSpark weakened its speculative-decoding value, while the auxiliary loss no longer justified its memory cost. 🔹 Heavily Compressed Attention: its global-summary role was ambiguous and difficult to combine with FP4 storage. 🔹 Dense warmup: V4.1 trains sparse attention from scratch rather than starting with one trillion dense-attention tokens. 2️⃣ Store less, reuse more Non-SWA KV cache moves from FP8 to FP4, while the more sensitive SWA portion remains FP8. DeepSeek no longer persists SWA cache. When a conversation forks from an earlier point, the system rebuilds only a small local window. Post-training simulated this process to limit numerical drift. A modified YOCO design provides the other major saving. Upper layers reuse the same lower-layer source representation, adding only a layer-specific projection. The result is roughly half the KV storage and close to 50% less historical prefill computation in the idealized case. 3️⃣ Sparse attention reuses its search Sparse attention lowers attention cost from O(n²) to O(kn), but finding the top-k tokens can still retain an O(n²) component. V4.1 Flash either reuses an earlier layer’s top-k result or selects a smaller candidate-block pool before re-indexing. This prevents token selection from becoming the bottleneck at long context lengths. 4️⃣ Engram and mHC were streamlined Engram replaces an expensive second-order optimizer state with a Sinkhorn-style update, removes causal convolution, and extends matching from 3-grams to 4-grams. mHC reorders residual mixing across layers, reducing estimated I/O from (4n+4)d to (3n+2)d. ✅ The larger pattern DeepSeek is not merely compressing an existing model. It is willing to discard its own previous components when a cheaper system-level design emerges. V4.1 Flash is less a smaller V4 than a new answer to one question: how much intelligence can be delivered per byte of memory and unit of inference cost? 🔗 Full analysis: #DeepSeek# #DeepSeekV41# #LLMArchitecture# #AIInfra# #KVCache# #SparseAttention#

An LLM's "residual connections" actually just add every layer with fixed, uniform weights ➕ As depth grows, each layer's contribution dilutes — this work answers that with attention. Title: Attention Residuals URL: ➕ Overview AttnRes replaces fixed-weight accumulation in residual connections with softmax attention, letting each layer selectively aggregate earlier representations using learned, input-dependent weights. ❓ Challenges Solved Modern LLMs use PreNorm residual connections as standard, but they sum layer outputs with fixed, uniform weights. ・Hidden states grow uncontrollably as depth increases ・Uniform aggregation progressively dilutes each layer's contribution In short, there was no control over which layer's representations get used, and how much. 💡 Methodology & Proposed Approach ・Full AttnRes: applies attention over all preceding layer outputs ・Block AttnRes: groups layers into blocks and attends only to block-level representations to cut compute ・Incorporates cache-based pipeline communication and a two-phase computation strategy ・Validated on the Kimi Linear architecture (48B total / 3B activated) 📊 Experimental Results ・Pre-trained on 1.4 trillion tokens ・Achieves more uniform output magnitudes and gradient distribution across depth ・Improves performance on all evaluated downstream tasks ・Scaling experiments confirm consistent gains across model sizes #LLM# #Architecture#

Human bottlenecks are why I'm skeptical about the sensationalized version of RSI being talked about in the policy and safety world right now. You can speed up, say, LLM architecture search by 1000x by removing humans and firehosing compute, but if that's 50% of your labor process (it isn't, ofc) and you only speed up the rest of the parts of the LLM production gantt chart by 2x, then your total speedup is ~4x. That means faster capabilities -- all else being equal -- but I'm not sure this is so different from other progress speed-ups we've found in the past due to compute (e.g. the huge unlock with RL/synthetic data replacing 2022-3 era pretraining+manually obtained SFT). And while some parts of the LLM production line are accelerating now and at any given moment, other parts of LLM development are attenuating now and at any given moment (e.g. growth in parameter count); all of this makes me skeptical we'll see the kind of idealized foom that keeps getting discussed. I also feel there are only a handful of folks in a position to give a confident take on this, they're working in secrecy, but there's a memeification of RSI happening where many folks who *aren't* in a position to know are reifying RSI as *the* central policy construct of Q3 2026. To be clear; there's a lot that can be automated and is being automated and we're learning to automate in LLM production. But claiming we're about to un-logjam the smoothnesss in labor productivity gains dictated by Amdahl's law feels like an unearned claim at the moment cc'ing @natolambert @thlarsen @tszzl @alexolegimas curious about yours and others' thoughts if you have time to respond

The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.