注册并分享邀请链接,可获得视频播放与邀请奖励。

netrunner
@plotarmordev
Cofounder & engineer. Running local AI, building things with it, and sharing what I learn.
加入 April 2022
785 正在关注    8.3K 粉丝
You can make your own NVFP4 quant now. Download the base weights, point your coding agent at an NVFP4 release you already run and trust, and tell it to quantize the same way. It doesn't take long. It's however not easy to know if it's a good quant! Benchmarks barely help. When comparing multiple 4bit Qwen3.8 27B quants, they often land within 1/2 points of BF16 on task accuracy. What they differ on KL divergence against the BF16 model, on speed, and on behavior: the NVFP4 ones tend to fall into reasoning loops. The tooling leads to issues too. A crashed vLLM eval run on Blackwell can still exit 0 and write plausible scores, which looks like a regression that isn't there. And a recent vLLM NVFP4 fix was for uninitialized scales producing NaN garbage, an engine bug that would look exactly like a bad quant. So don't trust a score alone. Check KL against BF16, run a long reasoning prompt to catch loops, and read the eval logs.
显示更多
0
15
49
3
转发到社区