Horace He(@cHHillee):One thing I'm kinda curious about is that a lot of inference providers seem to charge on *realized* kv cache hits? Tinker charges off of theoretical kv cache hits (based off of a 1 hour limit), which is more similar to how Anthropic charges based off of explicit TTLs for kv cache retention. Imo I think Tinker/Anthropic's approach makes a lot more sense? Under a "realized" kv cache hit model, inference engines getting more kv cache hits actually reduces their revenue... and I think we've seen some of this gaming on openrouter.

2026.10.28 16:23

One thing I'm kinda curious about is that a lot of inference providers seem to charge on *realized* kv cache hits? Tinker charges off of theoretical kv cache hits (based off of a 1 hour limit), which is more similar to how Anthropic charges based off of explicit TTLs for kv cache retention. Imo I think Tinker/Anthropic's approach makes a lot more sense? Under a "realized" kv cache hit model, inference engines getting more kv cache hits actually reduces their revenue... and I think we've seen some of this gaming on openrouter.

Zain@zainhas

2026.10.28 04:43

inference engines should be bending over backwards more to get cache hits really interesting direction: > An Internet for the KV Cache - sending a KV cache across the Internet can now be faster and cheaper than computing it again.