思维怪怪 (@0xLogicrw) — 推特瀏覽器

2026.08.10 04:14

过去两年，大模型推理优化一直在拿 KV Cache 开刀。它是模型读完上下文后留下的中间计算结果，内容越长，越吃显存和带宽。DeepSeek-V2 借助 MLA，相比 DeepSeek 67B 将 KV Cache 减少 93.3%；Kimi Linear 相比全 MLA 又最多减少 75%。大家都在让长上下文更便宜。但这些方案都有一道墙：缓存通常只能在同一个模型内部复用。Agent 从小模型切到大模型后，目标模型还得把前面几万甚至几十万 Token 重新计算一遍。模型路由越频繁，这笔重复计算越难忽略。英伟达最新论文开始拆这堵墙。团队发现，在同一家族、KV 结构匹配的不同大小模型之间，缓存存在明显的线性关系。用 500 段、每段 1024 Token 的文本校准一次，就能拟合出一套映射，把一个模型算好的 KV Cache 转给另一个模型继续用。 Qwen3-14B 切到 32B 时，32K 上下文重新计算约需 7 秒，转换缓存只要约 0.28 秒，快约 25 倍。换句话说，以后甚至可以先让小模型负责「读」：把长上下文算成 KV Cache；真正需要更强能力时，再把缓存转给大模型，让大模型直接开始「想」和生成。这会让模型路由省下更多算力。现在让简单任务跑小模型，主要省的是生成成本；一旦切到大模型，长上下文往往还得由大模型重新算一遍。跨模型 KV Cache 如果做成熟，连这部分 prefill 都能交给小模型完成。Agent 可以长期用便宜模型维护上下文，碰到难题才叫醒大模型，而且不用每次都从头「读档」。

Avi Chawla@_avichawla

2026.08.07 07:42

NVIDIA researchers did it again! They found a way to make KV cache transferable between models. The target model skips prefill entirely, and the conversion runs 2.7 to 25x faster than processing the context again. Let's understand why this is so important today. LLM APIs are stateless, so every turn sends the entire conversation back to the model. The model reads all of it again before writing a single new token, and all of it is billed as input. Prompt caching allows Anthropic and other providers to hold the KV cache for a stable prefix and bill a hit at roughly 10% of the base input rate, because the compute was already done once. The 90% reduction is one of the largest lever in LLM serving, which is why so much production work goes into keeping prefixes byte-stable. But the cache only works on the model that produced it. Keys and values are produced from that model's weights, so no other model can read them. In pratice, the constraint shows up in LLM routing. If the traffic is shifted to a different model for cost/capability reasons, the accumulated KV cache becomes invalid. As a result, the accumulated context has to be processed from scratch, and it's billed at full rate. NVIDIA's recent paper treats this as a representation problem. Prefill's only output is the KV cache, so to move KV between models, we need to convert one model's cache into the format the other expects. They first checked whether the conversion has any structure worth exploiting. They found that moving from Qwen3 14B to 32B, a plain linear regression from a single source layer reconstructed 56% of the variance in the target model's keys. The two models obviously may have different layer counts, so there is no natural one-to-one pairing between them. For each target layer they rank every source layer by how well it predicts that layer, then feed the top eight in together, which takes the reconstruction to 79%. The mapper itself has three parts: > Each target layer and head gets its own independent linear map, solved in one closed-form step rather than by gradient descent. > The cross-layer selection described above is the second part, and their ablation shows it carries the most weight of the three. > Keys also carry a position-dependent rotation from RoPE. They strip that rotation, fit the map in position-free space, then re-apply the target model's rotation at inference. Across six pairs from Qwen3, Llama 3.1 and Ministral 3, four retain 73 to 98% of the receiving model's standalone accuracy, and the conversion runs 3-25x faster than processing the context again. Prior work on cross-model KV reuse exists, but it either trains a neural adapter per pair or requires both models to be architecturally identical. This is probably the first version that is closed-form and training-free, so a lot of it is still open research. Every pair tested belongs to one family, so it works on Qwen to Qwen and Llama to Llama. Cross-family transfer is listed as future work. All six pairs mentioned above also happen to share KV head count and per-head dimension across scales. Mismatched head configurations are currently untested. The researchers scoped this to dense full-attention only, so sliding-window and attention-recurrent hybrids still need work. Here's the paper: Plenty of work is yet to be done. Still, the constraint being solved is genuine. Every model swap currently invalidates the full KV that was already paid for, and this is the first result showing that work might be recoverable without training anything extra. That said, all of this only matters because of what the KV cache is doing in the first place. I wrote a first-principles breakdown of it, covering why the model stores keys and values at all, why the cache grows with every token, and what generation speed looks like with and without it. Read it below.

0

1

0