Looking through key vLLM tickets. Probably time to update vLLM. 👇
v0.31.0 brings some practical improvements for local AI servers
⚡ Restart without reloading the weights.
vllm preload keeps model weights in GPU memory across engine restarts. Useful when tuning your serving setup. This also supports MTP draft models.
🚀 More speculative decoding options.
Model Runner V2 adds draft-model support, alongside async DFlash and DSpark adaptive verification for Gemma4. Benefits depend on your model and configuration.
🛠️ Fixes for stalled generation.
The release addresses a KV-cache/MTP deadlock and HiSparse issues involving speculative-token acceptance, prompt processing and caching.
🎛️ Better control when requests pile up.
New scheduling controls manage active requests and adapt how long prompts are processed.
Link in ALT