vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
가입 March 2024
37 팔로잉 중 54.1K 팬
4/ Together, TTFT drops nearly 70% at ~100K throughput.
Thanks to @deepseek_ai for the model and kernels, @nvidia for the collaboration, and @SemiAnalysis_ for AgentX. Built by @inferact and the vLLM community.
더 보기
