This new Nvidia paper is huge for model routing and the future of inference: re-using the KV cache across different LLMs. One of the greatest challenge in model routing during long-horizon agent settings is that the cache does not transfer across different models. As a result, naively switching models without KV-cache-awareness will result in wasting more money. Conversely, transferring the KV cache across models significantly expands the total amount of savings that can be achieved. This paper is a great step in that direction. Congrats to Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, @MMKamani7, @Ritika_Borkar, Makesh Tarun Chandran, Pantea Zardoshti, and Bita Darvish Rouhani. Two questions: 1. Do you plan to investigate cross-family transfer? I saw this in the future work section but curious to learn more about your intuitions on the opportunities and challenges here. 2. What was the reason for focusing mostly on small to large model transfers and not large to small?

whatever happened to our blue sky, song by delta9. all clips taken in Brum uk @IonisedSkyWatch @RohitMishra2024 @Demo2020cracy @ElizabethF6116 @Dervishizir @JonHTobiassen @paul_vk6 @LegendreKristy @druid10 @Tboo211 @Gp1ggy1 @Mylilangelsrose @nogps1 @LaShonLuv1