Ze-Wei (Johnny) Liou(@zwljohnny):Why do we store the SSM state at all? More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters. We only write it back every step so the next step can read it. ReplaySSM caches the recent inputs instead and rebuilds the state on the fly. Same outputs, half the memory traffic → ~2x on spec decode at large batch sizes, which barely even helped SSMs before → up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B) Work with @tri

Ze-Wei (Johnny) Liou

@zwljohnny

First year PhD @Princeton working on ML Systems

参加 February 2026

67 フォロー中 155 ファン

Ze-Wei (Johnny) Liou@zwljohnny

2026.06.15 13:44

Why do we store the SSM state at all? More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters. We only write it back every step so the next step can read it. ReplaySSM caches the recent inputs instead and rebuilds the state on the fly. Same outputs, half the memory traffic → ~2x on spec decode at large batch sizes, which barely even helped SSMs before → up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B) Work with @tri_dao Blog + Code👇