가입 후 초대 링크를 공유하면 동영상 재생 및 초대 보상을 받을 수 있습니다.

Benjamin Marie
@bnjmn_marie
Independent AI researcher (AI, LLM). Support my work:
가입 June 2019
227 팔로잉 중    7.7K 팬
I'm working on adding GGUF support to vLLM through the Humming kernels. In just 5 hours with GPT-6 Astra, I managed to get full support for quantization formats from IQ/Q2 to Q8, matching llama.cpp's speed in my benchmarks and outperforming it by ~30% at batch size 8. Still significantly slower than native Humming with GPTQ layers, though. A year ago, this would have taken me days, if not weeks, of work. What a time to be alive! Of course, it's still messy and limited in scope: Qwen3.5 architecture only Ampere GPUs tested (Grace Hopper and Blackwell support is implemented but untested) But it already achieves my only goal: significantly speeding up my GGUF evaluations. I'll definitely do this more often going forward. I might publish the code, but with no promises regarding maintenance.
더 보기