注册并分享邀请链接,可获得视频播放与邀请奖励。

vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
加入 March 2024
36 正在关注    50.2K 粉丝
762 commits. 315 contributors. 104 first-timers. vLLM v0.30.0 is live. 🎉 Highlights: 🤖 Hybrid-attention hot paths: Kimi K3 streamlines KDA, AttnRes, and MLA; DeepSeek-V4.1-Flash adds MXFP8 KV and async Engram; Qwen3.8-Flash-Next fuses QSA/PLE and cuts sparse-GQA overhead 🗄️ HiSparse adds a host tier beneath sparse-MLA decode; under GPU pressure, only top-k misses return to a per-request hot buffer 🛠️ Model Runner V2 brings EAGLE3-style drafts to pipeline parallelism and extends adaptive verification to every draft-model speculator through online acceptance estimation (#50514#, #52228#) 🖋️ Dual-key Gumbel-max watermark generation and detection, with per-request opt-out and speculative-decoding support 🆕 New models include GLM-5.3-Flash, K2-Horizon, Cohere Compass, and Bailing V3 VL ⚡ Fast Start keeps post-quantized, TP-sharded weights in a per-GPU daemon; restarts map them over CUDA IPC with --load-format ipc_cache Thread 👇
显示更多
0
17
223
30
转发到社区