Ze-Wei (Johnny) Liou
@zwljohnny
Second year PhD @Princeton advised by @tri_dao working on ML Systems
75 Following 394 Followers
ReplaySSM now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang!
It speeds up standard decoding (up to 1.47x) and speculative decoding (up to 2.1x), and now also enables a low-precision SSM state whose rounding error never propagates through the recurrence.
Building on our earlier ReplaySSM work, we collaborated with @NVIDIA to pin down what the method really unlocks. Caching SSM inputs instead of the state is what ReplaySSM does. What it changes is that computing the output and updating the state can be separated during decoding. All three gains follow from that separation.
Joint blog post, paper, and code with @NVIDIA and @tri_dao 👇
Show more
Why do we store the SSM state at all?
More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters.
We only write it back every step so the next step can read it.
ReplaySSM caches the recent inputs instead and rebuilds the state on the fly.
Same outputs, half the memory traffic
→ ~2x on spec decode at large batch sizes, which barely even helped SSMs before
→ up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B)
Work with @tri_dao
Blog + Code👇
Show more
