vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
参加 March 2024
37 フォロー中 54.1K ファン
4/ Together, TTFT drops nearly 70% at ~100K throughput.
Thanks to @deepseek_ai for the model and kernels, @nvidia for the collaboration, and @SemiAnalysis_ for AgentX. Built by @inferact and the vLLM community.
もっと見る
