DeepSeek V4.1 Flash Is an Architecture Reset, Not Just a Cheaper Model Despite having more total and active parameters than its predecessor, V4.1 Flash cuts working KV cache to one quarter and persistent cache storage to one eighth. Zhihu contributor Exhalation explains how it does this by deleting old modules, sharing KV states, and recomputing local context. 1️⃣ DeepSeek removed its own previous ideas 🔹 MTP: external draft models such as DSpark weakened its speculative-decoding value, while the auxiliary loss no longer justified its memory cost. 🔹 Heavily Compressed Attention: its global-summary role was ambiguous and difficult to combine with FP4 storage. 🔹 Dense warmup: V4.1 trains sparse attention from scratch rather than starting with one trillion dense-attention tokens. 2️⃣ Store less, reuse more Non-SWA KV cache moves from FP8 to FP4, while the more sensitive SWA portion remains FP8. DeepSeek no longer persists SWA cache. When a conversation forks from an earlier point, the system rebuilds only a small local window. Post-training simulated this process to limit numerical drift. A modified YOCO design provides the other major saving. Upper layers reuse the same lower-layer source representation, adding only a layer-specific projection. The result is roughly half the KV storage and close to 50% less historical prefill computation in the idealized case. 3️⃣ Sparse attention reuses its search Sparse attention lowers attention cost from O(n²) to O(kn), but finding the top-k tokens can still retain an O(n²) component. V4.1 Flash either reuses an earlier layer’s top-k result or selects a smaller candidate-block pool before re-indexing. This prevents token selection from becoming the bottleneck at long context lengths. 4️⃣ Engram and mHC were streamlined Engram replaces an expensive second-order optimizer state with a Sinkhorn-style update, removes causal convolution, and extends matching from 3-grams to 4-grams. mHC reorders residual mixing across layers, reducing estimated I/O from (4n+4)d to (3n+2)d. ✅ The larger pattern DeepSeek is not merely compressing an existing model. It is willing to discard its own previous components when a cheaper system-level design emerges. V4.1 Flash is less a smaller V4 than a new answer to one question: how much intelligence can be delivered per byte of memory and unit of inference cost? 🔗 Full analysis: #DeepSeek# #DeepSeekV41# #LLMArchitecture# #AIInfra# #KVCache# #SparseAttention#

Sparse Attention with Indexer also seems rather standard. Essentially, we use an indexer at block level in a compressed latent space. For training, the indexer is asked to predict the full attention score 1) Distill the dense attention scores into the indexer 2) KL loss on the indexer vs the full attention teacher

How GLM5.3 Sparse Attention Affects HBM Memory Usage GLM-5.3, KV Cache Offloading, HiSparse, AgentX TileRT, InferenceX DeepSeek Sparse Attention, IndexShare, Single-rollout Asynchronous Optimization, Cybersecurity

Long-context sparse attention has a catch: data-dependent block selection wrecks memory access kills speed. Our @MiniMax_AI M3 kernel on Blackwell answers it. KV-stationary, each block read once, ~980 TFLOP/s on a B200. See the breakdown here →

MSA (Memory Sparse Attention) represents our significant exploration in the field of long-term memory. It stands as the first end-to-end long-term memory framework for large models to genuinely achieve a 100M context length. Interestingly, as the memory length scales from 16K to 100M, the model's performance score decreases by a mere 9%, demonstrating highly robust scalability. Main contribution： 1，We propose MSA, an end-to-end trainable, scalable sparse attention architecture with a document-wise RoPE that extends intrinsic LLM memory while preserving representational alignment. It achieves near-linear inference cost and exhibits < 9% degradation even when scaling from 16K to 100M tokens. 2，We introduce KV cache compression to reduce memory footprint and latency while maintaining retrieval fidelity at scale. Paired with Memory Parallel, it enables high-throughput processing for 100M tokens under practical deployment constraints, such as a single 2×A800 GPU node. 3，We present Memory Interleave, an adaptive mechanism that facilitates complex multi-hop reasoning. By iteratively synchronizing and integrating KV cache across scattered context segments, MSA preserves cross-document dependencies and enables robust long-range evidence integration. 4，Comprehensive evaluations on long-context QA and Needle-In-A-Haystack benchmarks demonstrate that MSA significantly outperforms frontier LLMs, state-of-the-art RAG systems and leading memory agents. Welcome to feedback: We are looking for passionate talents to join our team! If you are interested in our work and vision, please don't hesitate to send us an email at evermind@shanda.com.

Tencent 🤨 📹dropped Prism -Opensource Video+Audio Gen Model -Dynamic Sparse Attention for 2K Video-Audio Generation -Supports native joint video-audio generation at 720p, 1080p, and 2K resolutions -Native joint video-audio training 👇

One difference between V4 and V4.1 papers is the details on sparse attention training. They say much more. We know concretely that they pretrain with 64K for 34T tokens, and then do 11T at 1M. No dense warm-up. No instabilities throughout.

Kimi K3.1 Pro weights are out. New: They replaced all 24 Gated MLA layers from K3 with Gated DeepSeek Sparse Attention. The other 69 KDA layers remain unchanged. DSA uses a top-k of just 2,048 for a model with 1M context. DeepSeek is a year ahead when it comes to architecture.