KV-Cache hits missing is the biggest TRADE-OFFS with these tools !! TLDR: Explained in the tldraw workflow image attached !! As soon as I saw this article, I went onto see if kv-cache has been talked about or not and apparently it was. It cites a paper which reads "KV-cache transfer in LLM families" - But if you would have given this paper a read anytime, it is actually not transfer but just REPRESENTATION TRANSFER" - There are several more studies on this but there is no solution to this yet, anytime you switch a model mid-task, KV cache will de optimise and you won't be getting the same results until you spend an enormous amount of tokens again to develop or tune the same context for the model and even after that some representations will differ and go off the birdge. - I have been working and researching in this space from few months now and this is the most interesting work I have been doing tbh. -As per what I have studied and found out with my study and research, SSMs are a great solution to this compared to LLMs. Simple and Concise explanation- - Instead of transferring KV cache state from: K_A, V_A to K_B, V_B for 2 different models, we will just have to transfer the nth state from SSMs. -And hence the state-n becomes the 1st state of the new model which has been switched. There is a lot more underlying concept and research to this and very less resource is available. Happy to have a conversation with people involved in KV CACHE and its transfer and topics around the similar corner.