DeepSeek V4.1 Gives Prefill and Decode Different Compute Paths Prompt tokens mostly traverse 20 layers; generated tokens traverse all 40. This Causal Encoder-Decoder design nearly halves long-input prefill while preserving autoregressive generation. Zhihu contributor 潜龙勿用, Changxin Ke(柯昌鑫), a graduate researcher at ICT, CAS, explains how its architecture and post-training were designed together. 1️⃣ A causal encoder, not T5 The 40-layer backbone is split into a 20-layer causal encoder and a 20-layer decoder. Both remain causal. The encoder processes the prompt and supplies the decoder’s global KV. Most prompt tokens avoid the decoder stack, while generated tokens still run through all 40 layers. That means 8B active parameters per prefill token versus 16B per decode token. 2️⃣ Most layers share global memory CSA2 uses three modes: Full creates global KV and an index; Reindex shares the KV but selects new positions; Reuse shares both. Only four layers create independent global KV, four reindex it, and 30 reuse both KV and the latest index. Each layer retains its own query and local SWA state. The first Full layer builds up to 16,384 candidates. Later layers search this pool for their Top-512. From 4K to 1M context, decode FLOPs per token rise only about 25%. 3️⃣ Serving approximations enter training Exact reconstruction of decoder SWA states would replay 2,560 prompt tokens. V4.1 replays only the final 128 encoder outputs and trains the model to tolerate the approximation. With FP4 global KV, cache falls to 890 bytes per token, persistent cache to roughly one eighth of V4-Flash, and prefill compute close to half. Bounded replay and constrained retrieval are not last-minute serving tricks. The model experiences them during post-training. 4️⃣ Post-training is an evolving Agent system Each RL task combines a problem, environment, and verifier. New trajectories can reveal shortcuts, broken environments, or verifier errors and send the task back for repair. V4.1 trains across multiple harnesses, merges checkpoints between RL runs, and finishes with on-policy distillation from more than 40 teachers. Raising reasoning effort from 25 to 100 increases output length about 2.5×, while average Pass@1 across eight benchmarks rises from 67.1% to 76.3%. ✅ The real design choice V4.1 aligns model structure, cache policy, retrieval limits, training environments, and inference around long-running Agents. The model learns under the same constraints the deployed system will actually impose. 🔗 Full analysis: #DeepSeek# #DeepSeekV41# #LLMArchitecture# #AIAgents# #LongContext# #AIInfra#

Long-context sparse attention has a catch: data-dependent block selection wrecks memory access kills speed. Our @MiniMax_AI M3 kernel on Blackwell answers it. KV-stationary, each block read once, ~980 TFLOP/s on a B200. See the breakdown here →

For long-context LLM inference, should the KV cache be offloaded to disk or just recomputed on the GPU? There's no universal right answer, and this paper builds a system that decides quantitatively. Title: Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs URL: 📝 Overview The paper proposes py-kvcache, an external KV cache system for vLLM. Using io_uring for async I/O, even a Python implementation pulls near-full SSD read bandwidth of 13.5GB/s. ❗ Problem it solves Existing external KV caches like LMCache have no criteria for when to actually use them — they load from disk unconditionally even when a short prefix or a fast GPU makes recomputation cheaper. ⚙️ Methodology It introduces "scheduler-aware preloading," which starts disk reads while a request is still waiting to be scheduled, plus a "break-even gate" that rejects a load whenever it wouldn't improve time-to-first-token. 📊 Results On LongBench multi-document QA it's 6.02-7.43x faster than GPU recomputation and 2.77-3.64x faster than LMCache. On multi-turn SCBench, native vLLM read 3.4TB from disk with completion times over 1200s, while py-kvcache kept disk reads to 85GB and completion time to 480s. 🖥️ Use cases On a high-end H100, requests often don't even clear the break-even point, so skipping external caching is fine — but on a lower-end RTX 4000 Ada, external caching clearly wins, giving concrete hardware-specific guidance. #LLMInference# #vLLM#

On benchmarking long-context agentic instruction following. Agent benchmarks mostly reward reaching the answer. This new benchmark measures whether the agent reached it the permitted way, which is the question enterprise deployments care about. If you ship skills files, policy documents, or long system prompts, you have been trusting that they actually bind agent behavior. But how are you measuring all of this? Surge AI built a benchmark to actually check this. HANDBOOK.md places a standard operating procedure of 20 to 124 pages in context and grades whether it governed every action across an extended tool-use horizon. 65 tasks, five domains, ten fictional companies. Each task runs in a self-contained company environment with a file workspace plus mock email, chat, calendar, issue-tracking, and commerce services exposed over MCP. Every task mutates one of ten base handbooks, altering the specific rules and thresholds that grading turns on, so memorization does not help. Grading is fully deterministic and two-sided. 824 programmatic criteria check that required actions occurred and that prohibited actions did not. Paper: Learn to build effective AI agents in our academy:

GEAR turns long-context grounding into a reward-shaping problem, by rewarding n-gram overlap with annotated evidence while penalizing overlap with distractors, reducing both copying and reasoning length. The method still depends on automatically generated evidence annotations and overlap proxies, but it gives long-context RL a concrete selectivity objective instead of another generic accuracy bonus. Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning Paper:

Understand entire hour-long videos and wield tools and search — an efficient multimodal model with 30B total params but only 3B active at inference 🎬 Title: Kwai Keye-VL-2.0 Technical Report URL: 🎬 Overview An open-source multimodal foundation model from Kuaishou, built for long-video understanding and agentic intelligence. It's a Mixture-of-Experts (MoE) model with 30B total parameters but only 3B activated at inference. ❓ Challenges Solved Processing hour-level videos demands enormous compute. ・Many frames make long-range temporal dependencies hard to capture ・The challenge was addressing that compute constraint while keeping strong performance across diverse tasks 💡 Methodology & Proposed Approach ・Long-context: adapts DeepSeek Sparse Attention (DSA) to GQA-based architectures for lossless 256K context processing, capturing key frames and long-range temporal dependencies ・Infrastructure: scalable video I/O, heterogeneous ViT-LM parallelism, custom DSA kernels ・Training: Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) with Context-RL and Video-RL to address catastrophic forgetting during multi-task alignment 📊 Experimental Results ・State-of-the-art among models of similar scale ・Especially strong on fine-grained temporal localization (TimeLens) ・Excels at long-video comprehension on Video-MME-v2 and LongVideoBench ・Also capable at multimodal agent collaboration across Code, Tool, and Search, with self-correction 🌍 Use Cases It fits long-video understanding, search, and moderation, plus backbones for video-handling autonomous agents. As the first application of sparse attention to multimodal at this scale, its big strength is making hour-level video processing cost-realistic. #VideoUnderstanding# #Multimodal#

💭 What if a model reading a long context already knows exactly where to look, but keeps dutifully re-scanning everything anyway? Long-context models have to sweep their entire KV cache at every decoding step, even though attention actually concentrates on a small handful of tokens. Existing sparse attention methods still need to re-search for those relevant tokens at every step, and that search itself becomes the bottleneck. Researchers at KAIST AI and Google DeepMind flipped the question: instead of scoring tokens externally, why not let the model declare where it's looking, in its own words? That's Declarative Attention. Inside its chain-of-thought, the model announces one of three modes -- for scanning everything, for a named segment, for just recent output -- and the inference engine turns that declaration directly into an attention mask. Remarkably, this works zero-shot, with no additional training. On Gemma-4-31B and Qwen-3.6-27B, attended tokens drop by 52.0% and 31.1% respectively, while accuracy only slips by 1-3 points. The effect gets more reliable as models and contexts grow larger, saving up to 21 million tokens per response on the longest cases. Language Models Can Control Their Own Attention By making attention patterns auditable while cutting inference cost, this looks like a real lever for lowering the cost of long-horizon agentic reasoning. #LLM# #Attention#