vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
参加 March 2024
36 フォロー中 50.2K ファン
vLLM integrated PyNvVideoCodec to offload video decoding from CPU to GPU's NVDEC.
Result: 2x+ throughput at 8×H100, CPU bottleneck gone! This is huge for video captioning at scale (AV training, metadata), and ships with CUDA vLLM releases.
🔗
もっと見る
