vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
๊ฐ์
March 2024
36 ํ๋ก์ ์ค 50.2K ํฌ
vLLM integrated PyNvVideoCodec to offload video decoding from CPU to GPU's NVDEC.
Result: 2x+ throughput at 8รH100, CPU bottleneck gone! This is huge for video captioning at scale (AV training, metadata), and ships with CUDA vLLM releases.
๐
๋ ๋ณด๊ธฐ
