What if Transformers had memory along the depth axis? This paper recasts residual connections as a regression problem. Title: Depth Memory URL: It unifies residuals, Attention Residuals and Hyper-Connections as Read/Write operations on memory streams, and proposes LDM (Linear Depth Memory), a fixed-size memory that stores information across layers. Three highlights: 🧮 An exact regression over depth Each layer's output is treated as a key-value observation, and a weighted least-squares problem with forgetting factors is solved with recursive least squares (RLS). The outer-product update of linear attention turns out to be just an approximation. ⚡ GPU implementation that makes it practical State unwinding recovers earlier states from a small saved context, and windowed writes batch updates. With CuTe kernels on B200, it beats Full AttnRes in throughput and is on par with Block AttnRes. 📈 Consistent gains at 400M and 1.5B At 1.5B, LAMBADA perplexity drops from 10.49 (PreLN) to 8.42 (LDM) with only about 1% extra parameters, and gains keep growing with depth. Removing the residual stream hurts a lot, a reminder of how much the additive path matters. #Transformer# #LLM#