Quantization is easy. Fast quantization kernels for every format × every GPU × every workload are not. Today we're open-sourcing Humming with Ant Group: one kernel library for all of it — up to 5x kernel speedups, +14–43% E2E MoE throughput. Blog + benchmarks:

Lossless quantization has usually meant giving up inference speedup. This paper changes that. SLQ (Statistically-Lossless Quantization) reaches task-lossless compression at 3.3 bits per parameter, and distribution-lossless at 5-6 bpp where the output distribution is practically indistinguishable from the original. 1.7 to 3.6x throughput over BF16 in @vllm_project. Beats FP8 while staying lossless. From Michael Helcig, @_EldarKurtic, and @DAlistarh.

Dozens of vector quantization papers come out every year. When we surveyed them, we noticed that most quantizers are not new algorithms. They are the same handful of primitive operations, strung together in a different order. So we built VQ-bench: an open-source library of those primitives, composable into pipelines. E-RaBitQ, one of the strongest methods we tested, is four primitives in a list. Swap one and you have a new quantizer, evaluated exactly the way every other method is evaluated. We used it to benchmark 14 popular quantizers on recall, reconstruction error, and encode time. Two takeaways so far: PQ and OPQ have the lowest reconstruction error, and EDEN matches E-RaBitQ on recall while encoding much faster. This is a first iteration. We want your feedback, corrections, and contributions, and we will keep adding quantizers over time. Website: Blog: Repo: Paper (VecDB@VLDB 2026):

“Why Does Post-Training Quantization Work?” This paper shows that pretrained LLMs are inherently robust to quantization because layers actively counteract accumulated quantization errors, while high-dimensional LM-head geometry preferentially preserves top-token predictions. This lets 4-bit models stay close to full precision despite substantial hidden-state perturbations.

Coming later today: statistically lossless quantization for Qwen3.8 27B, GGUF versions! > Better than the GPTQ versions published last week > Made from Unsloth's best UD GGUFs btw, when running this process, I found that Unsloth's UD Q8 K XL matches the original BF16 next-token full vocab probability distribution at 99.5%. That's insanely high and significantly better than the official FP8. Unbeatable.

All my oQe models have been updated with Improved oQe Quantization, available from oMLX 0.7.1. Compared with the previous oQe builds, KL divergence is 11.6–18.1% lower on code and 10.2–13.8% lower on prose, so please give them a try. The full model list is below. KL divergence from the original BF16 model (mean, lower is better) - Qwen3.8-27B oQ4e: code 0.0143 → 0.0117 (-18.1%), prose 0.0696 → 0.0600 (-13.8%) - Qwen3.8-Flash-Next oQ4e: code 0.0877 → 0.0776 (-11.6%), prose 0.1909 → 0.1713 (-10.2%) What's new - Below oQ4, the Engram and N-gram tables stay at 4 bits and are left out of the bit budget. oMLX can serve these tables from SSD instead of memory, so the full bit budget goes to the rest of the model. - Model cards now show the bit distribution and storage breakdown. If you downloaded an earlier oQe build, please re-download it. Improved oQe models (69) - Qwen3.8-Flash-Next: 2.5 · 2.7 · 3 · 3.5 · 4 · 6 · 8 bits - Qwen3.8-Flash-Next FP16: 2.5 · 3 · 3.5 · 4 bits - Qwen3.8-27B: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - Qwen3.6-35B-A3B: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - GLM-5.3-Flash: 2 · 2.5 · 3 · 3.5 · 4 · 5 · 6 bits - DeepSeek-V4.1-Flash: 2 · 2.5 · 3 · 4 bits - DeepSeek-V4-Flash-0731: 2.5 · 2.7 · 3 · 3.5 · 4 bits - gemma-4-31B-it: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - gemma-4-26B-A4B-it: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - gemma-4-12B-it: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - clef: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - clef-flash: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits

Today's YouTube video is another adventure in local AI and quantization: GSQ! In the video I try my best to explain the difference between GSQ and other quantization formats, and then actually test out a model in this format on my DGX Spark Check it out!

We’re now in the copy the HF quantization repo and put your name on it as an expert phase of Local AI That’s not how Opensource as a community flourishes You might have good intentions, but please do right by others & lift them up with you rather than falsely crediting yourself

Bonsai 2 has been evaluated with a low thinking budget for xhigh. Quantization errors really show their impact on long sequences, and Qwen3.8 27B often needs more than 81K tokens to complete its answer. For coding problems, like in LiveCodeBench, this is not enough. Expect some surprises for long-horizon agentic tasks. It's probably not as good as the model card says. Remarkable work nonetheless, as always.