alphaXiv(@askalphaxiv):"Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling" Linear attention is fast, but its fixed-size state limits memory. This paper expands that matrix into a 3D tensor using two keys, giving an 8x larger recurrent state with only ~1.2% more parameters. At 1.3B scale, it improves long-context recall and reaches 5.1x Transformer speed at 64k context.

2026.10.05 09:38

"Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling" Linear attention is fast, but its fixed-size state limits memory. This paper expands that matrix into a 3D tensor using two keys, giving an 8x larger recurrent state with only ~1.2% more parameters. At 1.3B scale, it improves long-context recall and reaches 5.1x Transformer speed at 64k context.