KVCache.AI(@KVCache_AI):🚀 Over one trillion tokens per day, powered by Mooncake and SGLang. @ApproachingAI shares the system architecture and engineering lessons behind its token factory, running leading trillion-parameter models with SGLang HiCache + Mooncake Store. A deep dive into: 🏗️ Production architecture and system design trade-offs ⚡ Performance optimizations for trillion-token-scale workloads 🛡️ Failure isolation, fault tolerance, and recovery Real production lessons on scaling KV cache reuse while meeting strict inference SLOs. Read more:

KVCache.AI

@KVCache_AI

Hi, this is official account. We build systems for efficient LLM serving, including KTransformers, Mooncake and AgentENV.

参加 August 2018

109 フォロー中 1.1K ファン

KVCache.AI@KVCache_AI

2026.09.15 11:17

🚀 Over one trillion tokens per day, powered by Mooncake and SGLang. @ApproachingAI shares the system architecture and engineering lessons behind its token factory, running leading trillion-parameter models with SGLang HiCache + Mooncake Store. A deep dive into: 🏗️ Production architecture and system design trade-offs ⚡ Performance optimizations for trillion-token-scale workloads 🛡️ Failure isolation, fault tolerance, and recovery Real production lessons on scaling KV cache reuse while meeting strict inference SLOs. Read more: