vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
加入 March 2024
37 正在關注 54.1K 粉絲
4/ Together, TTFT drops nearly 70% at ~100K throughput.
Thanks to @deepseek_ai for the model and kernels, @nvidia for the collaboration, and @SemiAnalysis_ for AgentX. Built by @inferact and the vLLM community.
顯示更多
