Ze-Wei (Johnny) Liou(@zwljohnny):Why do we store the SSM state at all? More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters. We only write it back every step so the next step can read it. ReplaySSM caches the recent inputs instead and rebuilds the state on the fly. Same outputs, half the memory traffic → ~2x on spec decode at large batch sizes, which barely even helped SSMs before → up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B) Work with @tri

Ze-Wei (Johnny) Liou

@zwljohnny

First year PhD @Princeton working on ML Systems

加入 February 2026

67 正在关注 155 粉丝

Ze-Wei (Johnny) Liou@zwljohnny

2026.06.15 13:44

Why do we store the SSM state at all? More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters. We only write it back every step so the next step can read it. ReplaySSM caches the recent inputs instead and rebuilds the state on the fly. Same outputs, half the memory traffic → ~2x on spec decode at large batch sizes, which barely even helped SSMs before → up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B) Work with @tri_dao Blog + Code👇

显示更多