Oliver Sieberling @ COLM(@osieberling):Linear attention is arguably the most naive RNN, but still massively outperforms traditional RNNs by maintaining a matrix state. So... what if we use a (triadic) outer product of three vectors and maintain a three-dimensional state? Introducing: Triadic Linear Attention 🧵

Oliver Sieberling @ COLM

@osieberling

PhD student @MIT, pretraining large neural networks, rnn enjoyer

가입 January 2024

794 팔로잉 중 950 팬

Oliver Sieberling @ COLM@osieberling

2026.10.04 21:15

Linear attention is arguably the most naive RNN, but still massively outperforms traditional RNNs by maintaining a matrix state. So... what if we use a (triadic) outer product of three vectors and maintain a three-dimensional state? Introducing: Triadic Linear Attention 🧵