This new Nvidia paper is huge for model routing and the future of inference: re-using the KV cache across different LLMs. One of the greatest challenge in model routing during long-horizon agent settings is that the cache does not transfer across different models. As a result, naively switching models without KV-cache-awareness will result in wasting more money. Conversely, transferring the KV cache across models significantly expands the total amount of savings that can be achieved. This paper is a great step in that direction. Congrats to Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, @MMKamani7, @Ritika_Borkar, Makesh Tarun Chandran, Pantea Zardoshti, and Bita Darvish Rouhani. Two questions: 1. Do you plan to investigate cross-family transfer? I saw this in the future work section but curious to learn more about your intuitions on the opportunities and challenges here. 2. What was the reason for focusing mostly on small to large model transfers and not large to small?

I'm giving a talk on model routing today at #Ai42026#. If managing AI inference spend is a priority for your organization, please drop by! I'll also be available after the talk for Q&A. @Ai4Conferences

Today we’re announcing Not Diamond Code, the world’s most powerful intelligent model router for long-horizon coding agents. Not Diamond works with any gateway or harness, including Claude Code, to select the best model and reasoning effort for each step, reducing costs by 20-65% without impacting quality.

Today we’re announcing Not Diamond Code, the world’s most powerful intelligent model router for long-horizon coding agents. Not Diamond works with any gateway or harness, including Claude Code, to select the best model and reasoning effort for each step, reducing costs by 20-65% without impacting quality.

Claude Code is the hegemony. Across the enterprises we work with, we see Claude Code dominating 80% of usage and spending. *But* I think this position is tenuous: - Enterprises often lag the frontier by 6 months. Claude Code's big moment was 6 months ago, so we'll see peak enterprise adoption now through eoy - OSS harnesses give enterprises more visibility, control, and choice across providers and models - The only defensibility for a harness is developer adoption, and individual developers are increasingly adopting codex, opencode, etc. Strong piece from @LauraBratton5 on this :

New article from @muradhem on Tokenomics. Enjoyed the interview! San Francisco-based firm Not Diamond also sells tools to cut clients’ AI bills. Its technology routes tasks from a coding agent to the AI system that will deliver the best response at the lowest possible cost. “A lot of people just default to the most powerful model for everything,” said CEO Tomás Hernando Kofman, but many tasks can be handled by lesser systems without sacrificing quality.

We've been working on intelligent model routing since 2023 and partner with many of the F100 to reduce their coding agent costs. It is a deceptively difficult problem with many pitfalls that people tend to miss until they fall right into them. When new models become even just a little bit better, they are able to run much longer autonomously / in parallel. This means that subtle variations in the model landscape will lead to major changes in your inference spend, even if model prices don't go up. Model routing helps ensure you are not overpaying for the entire course of additional workload volume at the price of the bleeding frontier. But naive approaches, most notably complexity and semantic classifiers, will fail in agentic settings and generally cost you *more* money because they are not cache-aware and they dumbly look at just the next turn without considering all downstream impacts of model recommendations over the full horizon of the trajectory. On top of this, they tend to be heuristically constructed, basically shifting the burden of manual model selection away from the user and to the opaque opinions of the router designer who is far from the front line. At Not Diamond, we approach model routing for coding agents in a more data-driven way that accounts for the multiple interrelated variables that drive cost. By doing so, we are able to consistently deliver 20-30%+ savings on coding agent workloads with no degradation relative to the frontier. We've written more about our approach, along with the shape of the problem space more generally, here: