註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Avi Chawla
@_avichawla
Daily tutorials and insights on DS, ML, LLMs, and RAGs • Co-founder @dailydoseofds_ • IIT Varanasi • ex-AI Engineer @ MastercardAI
加入 September 2019
152 正在關注    80.5K 粉絲
Prefix-aware LLM routing, clearly explained! (trick to get first LLM token 170x sooner) Production prompts start with the same system prompt, tool definitions and earlier turns, and only the last part of each request is new. So the GPU that served the previous turn still holds the KV cache for that shared prefix, but a cache-blind router may send the next request to a different GPU, which then has to recompute the whole prefix before it can write a single token. A prefix-aware router avoids this by tracking which GPU holds which part of each prompt. The visual below shows how it works: ↳ When a request arrives with prefix blocks A to D and a new suffix E, the router hashes the prompt block by block, so each hash covers everything before it. ↳ It looks those hashes up in a block-to-worker index, which stores locations and never the KV tensors themselves. ↳ Worker A holds 2 matching blocks, Worker C holds 3 and Worker B holds all 4, so Worker B gets the request. ↳ Worker B reuses the cached KV for A to D, computes KV only for E, and starts decoding. ↳ Once E is cached, a cache event updates the router's index, so the next turn lands on Worker B too. If no worker has a match, the request gets a full prefill on whichever worker the router picks. In a popular benchmark on 8 vLLM pods across 16 H100s, prefix-aware routing brought p90 time to first token down to 0.54s, while random routing took 92.5s on the same pods. Real routers also weigh load alongside cache matches, so the GPU with the warmest cache doesn't end up with the longest queue. All of this depends on the prefix cache itself working, and a small change early in the prompt can make every block after it miss. I covered how prefix caching works, and how prompt changes cause cache miss, in a detailed article. Read it below.
顯示更多
0
44
117
16
轉發到社區