注册并分享邀请链接,可获得视频播放与邀请奖励。

vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
加入 March 2024
36 正在关注    53.3K 粉丝
717 commits. 307 contributors. 96 first-timers. vLLM v0.31.0 is live. 🎉 Highlights: 🤖 DeepSeek-V4.1-Flash: FlashMLA mega attention with NVFP4 KV caching, DeepGEMM sparse MQA, Mega-Gate, and shared Engram host tables ⚡ Fast restart: vllm preload keeps weights in GPU memory across restarts, with data parallelism and MTP draft support; experimental engine snapshots 🛠️ Model Runner V2: draft-model speculative decoding, LiLiCorr, async DFlash, and DSpark adaptive verification for Gemma4 🌐 Large-scale serving: MoonEP balanced all-to-all, prefill context parallelism with data parallelism, DeepEPv2, and RL weight transfer 🎛️ Smarter scheduling: independent active-sequence limits, adaptive long-prefill chunking, and priority for requests already holding KV blocks 🗄️ HiSparse hardening: fixes for MTP verification, chunked-prefill livelocks, and host/GPU prefix caching Thanks to everyone who made this release possible! 💙 Full release notes 👇
显示更多