KV-Cache hits missing is the biggest TRADE-OFFS with these tools !! TLDR: Explained in the tldraw workflow image attached !! As soon as I saw this article, I went onto see if kv-cache has been talked about or not and apparently it was. It cites a paper which reads "KV-cache transfer in LLM families" - But if you would have given this paper a read anytime, it is actually not transfer but just REPRESENTATION TRANSFER" - There are several more studies on this but there is no solution to this yet, anytime you switch a model mid-task, KV cache will de optimise and you won't be getting the same results until you spend an enormous amount of tokens again to develop or tune the same context for the model and even after that some representations will differ and go off the birdge. - I have been working and researching in this space from few months now and this is the most interesting work I have been doing tbh. -As per what I have studied and found out with my study and research, SSMs are a great solution to this compared to LLMs. Simple and Concise explanation- - Instead of transferring KV cache state from: K_A, V_A to K_B, V_B for 2 different models, we will just have to transfer the nth state from SSMs. -And hence the state-n becomes the 1st state of the new model which has been switched. There is a lot more underlying concept and research to this and very less resource is available. Happy to have a conversation with people involved in KV CACHE and its transfer and topics around the similar corner.

AGENT EVALS is going to be a big thing and it’s a big white space to work on !! I have been working with agents from the past 6-8 months now and I have been shipping it in production pipelines as well. The thing about agent evals o have found out is - They don’t exist !! Yes, there is no proper eval that can be used in all the production grade agents or pipelines. Almost every big company has a blog on this - -Open Ai -Anthropic -Langchain -Langfuse -Data bricks -IBM -Hugging face articles So many repos as well but all are scope bounded. 2 days back I had to score few checkpoints in my agentic harness. How would I ? Simply prepared 6-8 mathematical scores and combined them with some ground truth and that became my agent eval for judgement. Now from the last 2 days any new model or pipeline or experimentation I do over those checkpoints, I can instantly scrape them on and off just via the scores. Simple and easy. Learn to build custom agent evals and you have a long way to go, the need for the same is gonna get even bigger !! For resources check this research paper or research book as you may call it: > GENERAL AGENT EVALUATION