Register and share your invite link to earn from video plays and referrals.

Avi Chawla
@_avichawla
Daily tutorials and insights on DS, ML, LLMs, and RAGs • Co-founder @dailydoseofds_ • IIT Varanasi • ex-AI Engineer @ MastercardAI
Joined September 2019
152 Following    80.5K Followers
A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.
Show more