I'm working on adding GGUF support to vLLM through the Humming kernels.
In just 5 hours with GPT-6 Astra, I managed to get full support for quantization formats from IQ/Q2 to Q8, matching llama.cpp's speed in my benchmarks and outperforming it by ~30% at batch size 8. Still significantly slower than native Humming with GPTQ layers, though.
A year ago, this would have taken me days, if not weeks, of work. What a time to be alive!
Of course, it's still messy and limited in scope:
Qwen3.5 architecture only
Ampere GPUs tested (Grace Hopper and Blackwell support is implemented but untested)
But it already achieves my only goal: significantly speeding up my GGUF evaluations. I'll definitely do this more often going forward.
I might publish the code, but with no promises regarding maintenance.