๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
๊ฐ€์ž… March 2024
36 ํŒ”๋กœ์ž‰ ์ค‘    53K ํŒฌ
717 commits. 307 contributors. 96 first-timers. vLLM v0.31.0 is live. ๐ŸŽ‰ Highlights: ๐Ÿค– DeepSeek-V4.1-Flash: FlashMLA mega attention with NVFP4 KV caching, DeepGEMM sparse MQA, Mega-Gate, and shared Engram host tables โšก Fast restart: vllm preload keeps weights in GPU memory across restarts, with data parallelism and MTP draft support; experimental engine snapshots ๐Ÿ› ๏ธ Model Runner V2: draft-model speculative decoding, LiLiCorr, async DFlash, and DSpark adaptive verification for Gemma4 ๐ŸŒ Large-scale serving: MoonEP balanced all-to-all, prefill context parallelism with data parallelism, DeepEPv2, and RL weight transfer ๐ŸŽ›๏ธ Smarter scheduling: independent active-sequence limits, adaptive long-prefill chunking, and priority for requests already holding KV blocks ๐Ÿ—„๏ธ HiSparse hardening: fixes for MTP verification, chunked-prefill livelocks, and host/GPU prefix caching Thanks to everyone who made this release possible! ๐Ÿ’™ Full release notes ๐Ÿ‘‡
๋” ๋ณด๊ธฐ