Looking through key vLLM tickets. Probably time to update vLLM. ๐
v0.31.0 brings some practical improvements for local AI servers
โก Restart without reloading the weights.
vllm preload keeps model weights in GPU memory across engine restarts. Useful when tuning your serving setup. This also supports MTP draft models.
๐ More speculative decoding options.
Model Runner V2 adds draft-model support, alongside async DFlash and DSpark adaptive verification for Gemma4. Benefits depend on your model and configuration.
๐ ๏ธ Fixes for stalled generation.
The release addresses a KV-cache/MTP deadlock and HiSparse issues involving speculative-token acceptance, prompt processing and caching.
๐๏ธ Better control when requests pile up.
New scheduling controls manage active requests and adapt how long prompts are processed.
Link in ALT