An LLM's "residual connections" actually just add every layer with fixed, uniform weights ➕ As depth grows, each layer's contribution dilutes — this work answers that with attention. Title: Attention Residuals URL: ➕ Overview AttnRes replaces fixed-weight accumulation in residual connections with softmax attention, letting each layer selectively aggregate earlier representations using learned, input-dependent weights. ❓ Challenges Solved Modern LLMs use PreNorm residual connections as standard, but they sum layer outputs with fixed, uniform weights. ・Hidden states grow uncontrollably as depth increases ・Uniform aggregation progressively dilutes each layer's contribution In short, there was no control over which layer's representations get used, and how much. 💡 Methodology & Proposed Approach ・Full AttnRes: applies attention over all preceding layer outputs ・Block AttnRes: groups layers into blocks and attends only to block-level representations to cut compute ・Incorporates cache-based pipeline communication and a two-phase computation strategy ・Validated on the Kimi Linear architecture (48B total / 3B activated) 📊 Experimental Results ・Pre-trained on 1.4 trillion tokens ・Achieves more uniform output magnitudes and gradient distribution across depth ・Improves performance on all evaluated downstream tasks ・Scaling experiments confirm consistent gains across model sizes #LLM# #Architecture#