KVCache.AI
@KVCache_AI
Hi, this is official account. We build systems for efficient LLM serving, including KTransformers, Mooncake and AgentENV.
가입 August 2018
109 팔로잉 중 1.1K 팬
TileRT is pushing NVIDIA GPUs into a new regime of ultra-high-interactivity inference, with a specialized decode engine delivering hundreds of tokens per second per user.
In its disaggregated prefill/decode architecture, Mooncake Transfer Engine provides the high-performance data path for moving KV cache from vLLM prefill nodes to TileRT decode nodes.
Read the full deep dive from SemiAnalysis:
#LLMInference# #TileRT# #vLLM# #Mooncake#
더 보기
