登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

Benjamin Marie
@bnjmn_marie
Independent AI researcher (AI, LLM). Support my work:
参加 June 2019
227 フォロー中    7.7K ファン
All these numbers are correct. "incorrect" is not how I would call this. That really looks like a setting mismatch. I can see these kinds of difference for LiveCodeBench when running it with different sandboxes. Unsloth LiveCodeBench numbers, for all the models evaluated here, are quite low. Almost Qwen3 32B level (what I reached with EvalScope 1.8.1's sandbox and rules). AIME25 numbers are also low but again it depends on what kind of parsing rules we use, so it's possible. That's why we should never compare our own numbers with previously published numbers. Have to rerun everything, like Unsloth did.
もっと見る
We're releasing new Unsloth v3.1 GGUF quants for DeepSeek-V4.1-Flash-1017. We ran benchmarks and v3 + v3.1 do better than GSQ-RCO. We also cannot reproduce numbers ran by GSQ-RCO - AIME25 for 1bit is not 100% but rather 91.88%. Livecodebench and GPQA are similarly incorrect. There are also claims that KLD is not a good metric, and that people can "game" KLD. The pirmary goal of quantization is to minimize KLD but also minimize disk space. If you get a KLD of 0, this means your quantized model == 16bit model, as logits are perfectly modelled. We ran benchmarks on multiple benchmarks including AIME25, Aider Polyglot, LiveCodeBench v6, HMMT 2026, GPQA Diamond, MMLU-Pro and IFEval, and also recorded reasoning length to see if lower bit quants think for longer. In we show that lower bit quants do think somewhat longer than the unquantized model - we plan to investigate and make quants even better. However, we're still much ahead vs GSQ-RCO on even reasoning length as well. We provide full benchmarking data, benchmarking methods (for eg we ran AIME25 16 times per question and took an average) and more in our updated blog at
もっと見る