๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
๊ฐ€์ž… March 2024
37 ํŒ”๋กœ์ž‰ ์ค‘    54.1K ํŒฌ
vLLM v0.33.0 is out! ๐Ÿš€ 927 commits ยท 389 contributors ยท 122 first-timers ๐Ÿค– New models: Reflection Beam-1.0, Step-5 Preview, Nemotron Parse 3.5, MiniCPM-OCR 1.0 ๐Ÿ”ฅ PyTorch 2.14 โ€” CUDA 13.0-only wheels, libtorch ABI-stable ๐ŸŽฒ Spec decoding: XPress refiner, FLy lossy verify (1.04โ€“1.31x), DSpark 1373โ†’3390 tok/s on GB300 โšก NVIDIA decode: 1.5โ€“2x latency decode on GB300, FA4 sparse MLA, 39โ€“48% lower TTFT ๐Ÿš€ New AWS EFA image + shorter vllm/vllm name ๐Ÿง ROCm 10.0 default, MonoKernel 2.32x P50 Full notes โžก๏ธ
๋” ๋ณด๊ธฐ