登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

Benjamin Marie
@bnjmn_marie
Independent AI researcher (AI, LLM). Support my work:
参加 June 2019
227 フォロー中    7.7K ファン
I'm working on adding GGUF support to vLLM through the Humming kernels. In just 5 hours with GPT-6 Astra, I managed to get full support for quantization formats from IQ/Q2 to Q8, matching llama.cpp's speed in my benchmarks and outperforming it by ~30% at batch size 8. Still significantly slower than native Humming with GPTQ layers, though. A year ago, this would have taken me days, if not weeks, of work. What a time to be alive! Of course, it's still messy and limited in scope: Qwen3.5 architecture only Ampere GPUs tested (Grace Hopper and Blackwell support is implemented but untested) But it already achieves my only goal: significantly speeding up my GGUF evaluations. I'll definitely do this more often going forward. I might publish the code, but with no promises regarding maintenance.
もっと見る