Oliver Sieberling @ COLM
@osieberling
PhD student @MIT, pretraining large neural networks, rnn enjoyer
794 Following 950 Followers
👀
This is pretty cool, less expressive than Linear version of 2-simplicial, as the product is QK1 hadamard QK2 and rank can be higher than each. Would be cool to see QK1K2 V1V2 variant
"Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling"
Linear attention is fast, but its fixed-size state limits memory.
This paper expands that matrix into a 3D tensor using two keys, giving an 8x larger recurrent state with only ~1.2% more parameters.
At 1.3B scale, it improves long-context recall and reaches 5.1x Transformer speed at 64k context.
Show more
Efficient way to grow the state size w.r.t params 🔥
Linear attention is arguably the most naive RNN, but still massively outperforms traditional RNNs by maintaining a matrix state. So... what if we use a (triadic) outer product of three vectors and maintain a three-dimensional state?
Introducing: Triadic Linear Attention 🧵
Show more
(1/7) Humans can acquire many capabilities without (catastrophically) forgetting previous ones. Why can't neural networks do the same?
Local Support Learning (LSL) takes us a step closer towards this goal.
In LLMs of up to 7B parameters, LSL augments gradient-based training to retain prior capabilities while learning new tasks at full capacity. Notably, it does so without access to prior data.
Key idea: casting forgetting as a geometric problem, and optimizing for the worst case.
Paper:
Website and code:
Show more
There’s lots of buzz around agentic harnesses for robots, particularly for long horizon tasks that require complex reasoning and memory. But what will it really take to turn a reasoning agent into a reactive, reliable and low-latency robot policy?
In our new paper, workspace models, we design a new memory architecture that acts as a latent harness for stronger reasoning models. Here’s why we think this might be the way forward:
(1/9)
Show more
Disaggregated quantization for LLMs 🧵
LLM prefill and decode can be processed by separate weights in different precisions. Boosts speed AND accuracy. Offloading makes it zero-overhead on long sequences for dense models. Applicable to existing quantized checkpoints.
Paper:
Code:
Qwen3.8 NVFP4 prefillers:
Show more
ScaleML will be back on Thursday! Join the mailing list on
Scale ML is resuming this Fall with Alok from Jane Street!
We will go over how to do data mixture experiments at scale! 🔥
Sign up for the zoom link via the website.
Scale ML is resuming this Fall with Alok from Jane Street!
We will go over how to do data mixture experiments at scale! 🔥
Sign up for the zoom link via the website.
