KVCache.AI
@KVCache_AI
Hi, this is official account. We build systems for efficient LLM serving, including KTransformers, Mooncake and AgentENV.
加入 August 2018
109 正在關注 1.1K 粉絲
TileRT is pushing NVIDIA GPUs into a new regime of ultra-high-interactivity inference, with a specialized decode engine delivering hundreds of tokens per second per user.
In its disaggregated prefill/decode architecture, Mooncake Transfer Engine provides the high-performance data path for moving KV cache from vLLM prefill nodes to TileRT decode nodes.
Read the full deep dive from SemiAnalysis:
#LLMInference# #TileRT# #vLLM# #Mooncake#
顯示更多
