vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
Joined March 2024
36 Following 50.2K Followers
vLLM integrated PyNvVideoCodec to offload video decoding from CPU to GPU's NVDEC.
Result: 2x+ throughput at 8×H100, CPU bottleneck gone! This is huge for video captioning at scale (AV training, metadata), and ships with CUDA vLLM releases.
🔗
Show more
