Lossless quantization has usually meant giving up inference speedup. This paper changes that. SLQ (Statistically-Lossless Quantization) reaches task-lossless compression at 3.3 bits per parameter, and distribution-lossless at 5-6 bpp where the output distribution is practically indistinguishable from the original. 1.7 to 3.6x throughput over BF16 in @vllm_project. Beats FP8 while staying lossless. From Michael Helcig, @_EldarKurtic, and @DAlistarh.

Dozens of vector quantization papers come out every year. When we surveyed them, we noticed that most quantizers are not new algorithms. They are the same handful of primitive operations, strung together in a different order. So we built VQ-bench: an open-source library of those primitives, composable into pipelines. E-RaBitQ, one of the strongest methods we tested, is four primitives in a list. Swap one and you have a new quantizer, evaluated exactly the way every other method is evaluated. We used it to benchmark 14 popular quantizers on recall, reconstruction error, and encode time. Two takeaways so far: PQ and OPQ have the lowest reconstruction error, and EDEN matches E-RaBitQ on recall while encoding much faster. This is a first iteration. We want your feedback, corrections, and contributions, and we will keep adding quantizers over time. Website: Blog: Repo: Paper (VecDB@VLDB 2026):

“Why Does Post-Training Quantization Work?” This paper shows that pretrained LLMs are inherently robust to quantization because layers actively counteract accumulated quantization errors, while high-dimensional LM-head geometry preferentially preserves top-token predictions. This lets 4-bit models stay close to full precision despite substantial hidden-state perturbations.

We’re now in the copy the HF quantization repo and put your name on it as an expert phase of Local AI That’s not how Opensource as a community flourishes You might have good intentions, but please do right by others & lift them up with you rather than falsely crediting yourself

Bonsai 2 has been evaluated with a low thinking budget for xhigh. Quantization errors really show their impact on long sequences, and Qwen3.8 27B often needs more than 81K tokens to complete its answer. For coding problems, like in LiveCodeBench, this is not enough. Expect some surprises for long-horizon agentic tasks. It's probably not as good as the model card says. Remarkable work nonetheless, as always.

benchmarks of a 50% pruned Qwen3.6-35b-a3b and expert-specific quantization technique (made by me) 7.3gb model preforming => 51gb model, exiting to see where I can bring this technique to. I have some more things lined up too. I need a DGX spark😭

legendary drop from NVIDIA: ModelOpt 0.45.0 biggest additions: - New NVFP4 (W4A16) weight-only quantization format that requires no calibration - Better MoE support (including mixed NVFP4 + FP8 recipes for models like Nemotron) - Easy MXFP4 → NVFP4 conversion for models like DeepSeek V4 and GPT-OSS - Various improvements for large-scale PTQ and Megatron workflows looks like NVIDIA is pushing harder on making 4-bit inference more practical and calibration-free.

After seeing so many good example results out of the MiMo v2.6 Pro RL I setup a recipe using Jarrelscy AVRQ hybrid quantization which is the first time i’m using this type of quantization on Four DGX-sparks with sm-121 since Jarrelscy recipe is for SM120. Adapted and wrote a script for it to work on the SM121 GB10 stack . Thanks @jarrelscy for the quantization/vllm sm120 Now you can run the MiMo V2.6 PRO-RL on TP=4 on 4 DGX sparks with 1M context and 2M KV token pool Dflash and Ablit coming for this. Stay tuned

Pinecone introduces VQ-bench, an open-source framework for testing and comparing vector quantization methods. See how it makes quantizers easier to build, benchmark, and evaluate across metrics like recall, reconstruction error, and performance. Read the full breakdown on the Pinecone blog:

Timelapse #14#; 7 hours (9:00am-4:00pm) What I did: - Started experimenting with quantization aware REAP - Studied online post-training techniques (DPO, PPO, GRPO) - Tested qwen3.8 through ambassador program - benched qwen3.8 @neutralityproject - Made 2 synthetic datasets for left/right output of LLM’s - Stretched every hour + posture excises and rice bucket training - Watched rekraps newest video - Watched Martin Shkreli stream - Scrolled X Song: Memory limitations in artificial intelligence - Infinity Frequencies