๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

David Hendrickson
@TeksEdge
CEO & Founder | PhD | Startup Advisor | @Columbia | Author Generative Software Engineering | ๐Ÿ”” Follow for AI & Vibe Coding Tips ๐Ÿ‘‡
๊ฐ€์ž… July 2023
552 ํŒ”๋กœ์ž‰ ์ค‘    12.5K ํŒฌ
Looking through key vLLM tickets. Probably time to update vLLM. ๐Ÿ‘‡ v0.31.0 brings some practical improvements for local AI servers โšก Restart without reloading the weights. vllm preload keeps model weights in GPU memory across engine restarts. Useful when tuning your serving setup. This also supports MTP draft models. ๐Ÿš€ More speculative decoding options. Model Runner V2 adds draft-model support, alongside async DFlash and DSpark adaptive verification for Gemma4. Benefits depend on your model and configuration. ๐Ÿ› ๏ธ Fixes for stalled generation. The release addresses a KV-cache/MTP deadlock and HiSparse issues involving speculative-token acceptance, prompt processing and caching. ๐ŸŽ›๏ธ Better control when requests pile up. New scheduling controls manage active requests and adapt how long prompts are processed. Link in ALT
๋” ๋ณด๊ธฐ