Alec Helbling
@alec_helbling
Interpretability, Multimodality, Diffusion. PhDing @GeorgiaTech. NSF Fellow. Prev intern @Apple, @Adobe, @NASAJPL.
2.1K Following 11.5K Followers
KV caching is the fundamental optimization underpinning autoregressive LLM inference.
Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated.
This avoids recomputation, but at the cost of memory capacity and bandwidth.
Show more
