Agent workloads bring long contexts, bursty traffic, and frequent tool calls, creating new demands for AI serving. Together with @AMD , we rebuilt the stack for Kimi K2.6 on AMD Instinct™ MI355X, with scheduler-aware multi-tier KV caching. Up to 3.2× smaller p99 TTFT and 7.7% higher total-token throughput, with no accuracy tax. Happy to see more Kimi running on AMD chips! Tech blog:

AGENTIC TRAFFIC NOW MAKES UP MORE THAN 70% OF ALL INFERENCE TRAFFIC 🚀 Agentic workloads are characterized by four elements: 🟠 Multi-turn: a session includes tens or hundreds of turns, leading to high potential KV-cache reuse. 🟠 Long context: system prompts, tool definitions, and the large number of turns make context accumulate quickly. 🟠 High prefix reuse: since the conversation progresses linearly, where output from turn n-1 is concatenated to turn n (typically), most context can be served from KV cache rather than recomputed (this depends on the amount of storage available to store KV tensors). As n grows, the ratio of cached input relative to uncached input typically tends towards 1. 🟠 Sub-agent bursts: a session launches multiple short-lived sub-agents with fresh context, which create bursty KV-cache patterns.