KVCache.AI
@KVCache_AI
Hi, this is official account. We build systems for efficient LLM serving, including KTransformers, Mooncake and AgentENV.
加入 August 2018
109 正在关注 1.1K 粉丝
TileRT is pushing NVIDIA GPUs into a new regime of ultra-high-interactivity inference, with a specialized decode engine delivering hundreds of tokens per second per user.
In its disaggregated prefill/decode architecture, Mooncake Transfer Engine provides the high-performance data path for moving KV cache from vLLM prefill nodes to TileRT decode nodes.
Read the full deep dive from SemiAnalysis:
#LLMInference# #TileRT# #vLLM# #Mooncake#
显示更多
