注册并分享邀请链接,可获得视频播放与邀请奖励。

alphaXiv
@askalphaxiv
High fidelity research
加入 November 2023
106 正在关注    61K 粉丝
"Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling" Linear attention is fast, but its fixed-size state limits memory. This paper expands that matrix into a 3D tensor using two keys, giving an 8x larger recurrent state with only ~1.2% more parameters. At 1.3B scale, it improves long-context recall and reaches 5.1x Transformer speed at 64k context.
显示更多
0
6
119
18
转发到社区