Agent workloads bring long contexts, bursty traffic, and frequent tool calls, creating new demands for AI serving. Together with @AMD , we rebuilt the stack for Kimi K2.6 on AMD Instinct™ MI355X, with scheduler-aware multi-tier KV caching. Up to 3.2× smaller p99 TTFT and 7.7% higher total-token throughput, with no accuracy tax. Happy to see more Kimi running on AMD chips! Tech blog:

20,000 Substack subscribers. Thank you all for the support over the years! I have been writing for nearly 3 years now, and I have learned a lot along the way in an unprecedented time for semiconductor hardware. Whether you are on a free tier or paid tier, thank you for opening up your email inbox to me in such a noisy world. I have exactly published 200 articles as of today. It's satisfyingly easy to say I get a 100 free subs per post, on average... in reality, its always bursty though. I will continue to do my best to write articles so that I can provide deep semi insights.. Cheers! 🍻

AGENTIC TRAFFIC NOW MAKES UP MORE THAN 70% OF ALL INFERENCE TRAFFIC 🚀 Agentic workloads are characterized by four elements: 🟠 Multi-turn: a session includes tens or hundreds of turns, leading to high potential KV-cache reuse. 🟠 Long context: system prompts, tool definitions, and the large number of turns make context accumulate quickly. 🟠 High prefix reuse: since the conversation progresses linearly, where output from turn n-1 is concatenated to turn n (typically), most context can be served from KV cache rather than recomputed (this depends on the amount of storage available to store KV tensors). As n grows, the ratio of cached input relative to uncached input typically tends towards 1. 🟠 Sub-agent bursts: a session launches multiple short-lived sub-agents with fresh context, which create bursty KV-cache patterns.