Register and share your invite link to earn from video plays and referrals.

Benjamin Marie
@bnjmn_marie
Independent AI researcher (AI, LLM). Support my work:
226 Following    7.5K Followers
This is one of those situations where I wish I had B200s to recompute these scores, along with an older model like Qwen3 32B Because these quants really seem as good as Qwen3 32B on these benchmarks...
Show more
That's quite a gap! Thanks unsloth for doing these evals I wonder how this correlates with EAR. This I might be able to run, but ultimately, task accuracy is what matters
GPT 6.1 Sol is such an annoying model. Yes, it is quite capable and cheap. But it's very lazy and needs to be pushed constantly. Like I'm wondering whether I should not set up a local Qwen3.8 27B to just push it to work harder every time it stops a task.
Show more
We have noticed recent comparisons published by Unsloth that do not align with our results for DeepSeek-V4.1-Flash-1017. So just if it is not obvious: we did of course not modify any hyperparameters to favor our models. We used standard settings appropriate for a model of this scale (763B parameters) and its 1M context length capability: a 128K context window and a reasoning effort set to 50. To the best of our knowledge, the discrepancy in performance stems from a difference in benchmark constraints - not in the model being worse. Unsloth's comparisons has a strict 32K token reasoning budget, whereas our benchmarks allow up to 128K output tokens. This significant difference in compute explains the variance in results; it is not due to any hidden adjustments on our part. Furthermore, imposing such a low reasoning budget (32K) appears inconsistent for a model designed with a 1M context window and 763B params. Our primary goal is open collaboration and transparency. All of our code is available on GitHub, and our papers are on arXiv. We are genuinely happy that other quantizations are performing well, as this was a key motivation for publishing these models in the first place. We invite anyone with the compute resources to reproduce our results using our exact settings (128K output budget, reasoning effort 50) for independent verification. Our previous models were benchmarked by third parties like @bnjmn_marie, who do great work for the community. We welcome rigorous testing of our claims. Furthermore, we are currently running the comparisons on the models within our setup and will post results here soon. Thanks and stay tuned!
Show more
All these numbers are correct. "incorrect" is not how I would call this. That really looks like a setting mismatch. I can see these kinds of difference for LiveCodeBench when running it with different sandboxes. Unsloth LiveCodeBench numbers, for all the models evaluated here, are quite low. Almost Qwen3 32B level (what I reached with EvalScope 1.8.1's sandbox and rules). AIME25 numbers are also low but again it depends on what kind of parsing rules we use, so it's possible. That's why we should never compare our own numbers with previously published numbers. Have to rerun everything, like Unsloth did.
Show more
We're releasing new Unsloth v3.1 GGUF quants for DeepSeek-V4.1-Flash-1017. We ran benchmarks and v3 + v3.1 do better than GSQ-RCO. We also cannot reproduce numbers ran by GSQ-RCO - AIME25 for 1bit is not 100% but rather 91.88%. Livecodebench and GPQA are similarly incorrect. There are also claims that KLD is not a good metric, and that people can "game" KLD. The pirmary goal of quantization is to minimize KLD but also minimize disk space. If you get a KLD of 0, this means your quantized model == 16bit model, as logits are perfectly modelled. We ran benchmarks on multiple benchmarks including AIME25, Aider Polyglot, LiveCodeBench v6, HMMT 2026, GPQA Diamond, MMLU-Pro and IFEval, and also recorded reasoning length to see if lower bit quants think for longer. In we show that lower bit quants do think somewhat longer than the unquantized model - we plan to investigate and make quants even better. However, we're still much ahead vs GSQ-RCO on even reasoning length as well. We provide full benchmarking data, benchmarking methods (for eg we ran AIME25 16 times per question and took an average) and more in our updated blog at
Show more
That's quite a gap! Thanks unsloth for doing these evals I wonder how this correlates with EAR. This I might be able to run, but ultimately, task accuracy is what matters
We’re releasing new DeepSeek-V4.1-Flash-1017 GGUFs, ranking highest across benchmarks. Dynamic V3.1 leads across nearly all benchmarks: GPQA Diamond, AIME25, LiveCodeBench v6, MMLU-Pro, Aider & more. Run on 64GB+ RAM. GGUF: Blog:
Show more
New release: Two smaller Qwen3.8 27B SLQ models! - A new one targeting EAR 98 - And another one (EAR 97.8) just aiming to beat the official FP8 while being ~10 GB smaller. This time, I searched with smaller group sizes, especially for INT2/INT3, but none were selected. Smaller groups at higher precision are more useful here. GPTQ just isn't good enough at very low precision. I'll try smarter (and more expensive...) quantization for Qwen4. These models using smaller groups could also be slightly slower than previous versions. I didn't check. I think we're close to the limit of what GPTQ can achieve at EAR 98. The models could be even smaller with quantized token embeddings, but a vLLM bug prevents this for Qwen3.5. I avoided relying on a patched vLLM to keep deployment simple. Models:
Show more
Many seem to think that the announced Qwen4 models will be open-weight We will probably only get the 27B I'm not sure we will get flash
Doubling or not halving?? Nothing changes right?
Day 25/ We are doubling all usage limits for all Pro 100/200/500 tiers and Business Premium, effective today. We heard your kind feedback and won’t be reducing usage limits for Pro 200, so your Work and Codex usage will stay the same. We’re also going to continue offering extended limits for dots for now, and usage of your dot will still continue to be separate from your default usage. And we'll of course also keep shipping better models and other efficiency improvements that increase your effective usage over time.
Show more
GPT 6 Astra is AGI but it still duplicates CSS style sheets, one for each page... and then forget to update them.
So we will never see GPT OSS 2
sam: local inference is a terrible allocation of scarce resources “a lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data centre but from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment somehow many people's intuition is the opposite of that”
Show more
it really looks much better than the nerfed Muse Open. Slightly larger, but you can try the GGUFs made by Unsloth. UD-Q2_K_XL might still be good enough. (I won't test them, still too large for my GPUs)
Show more
Today, we release the weights of a new and improved Le Chonk, advancing the open frontier for agentic coding and cybersecurity. 💻 Built to build: Le Chonk ranks #1# on @cognition’s FrontierCode among open models, and sets a new benchmark for Pareto efficiency on @artificialanalysis. 🛡 Built to defend: State-of-the-art in cyberdefense, put through the paces by the world's leading cyber vendors, including @CrowdStrike, @cloudflare, @rubrik, @zscaler, and others. Built for sovereignty: Make it yours. Download the weights on @huggingface , customize it with Mistral Forge, and deploy it wherever you need it. Read more from our team and customers 👇:
Show more
Today I'll release: - much smaller statistically lossless checkpoints of Qwen3.8 27B, with EAR passing 98, for vLLM (should work also with SGLang but I didn't test). => around 19 GB while still being better than the official FP8 - the smallest LFM3 1.7B GGUF with an EAR passing 98 (this was super hard!! very sensitive to quantization this one) I'm also working on a post-SLQ refinement step, using autoround. It already works but it's quite costly. I will probably release these v2 this weekend. GSQ would also be a good refinement steps for 2-bit and 3-bit layers. I'm checking the cost. I'm almost ready for Qwen4 🫡
Show more
I couldn't agree more! Kld is only used to evaluate quantized models because it's very cheap. It was not designed for LLM eval. And most GGUF users, who dont have any ml background, have no idea what it means and over interpret it Still good for quick diagnostic purposes
Show more
Faster nvfp4 decoding 👀
PyTorch 2.15 is here, with 3,117 commits from 583 contributors and updates across distributed training, NVGEMM, Apple Silicon, and accelerator platform support. The release makes the rewritten NCCL backend the default and adds one-sided transports with a NIXL backend. FSDP2 and pipelining gain finer memory and communication control. It also brings faster 4-bit (NVFP4) LLM decoding on NVIDIA Blackwell GPUs, FP8 and faster decode on Apple Silicon, and broader platform support. On Thursday, November 5, at 11 AM PST, join Tristan Rice (@rice_fry), Natalia Gimelshein, Huy Do, and Jane Xu (@janeyx99) (@Meta) for a live PyTorch 2.15 Q&A, moderated by Chris Gottbrath (Gottbrath Tech). The speakers will share an overview of the release and answer community questions about PyTorch and the new capabilities in this release. 🔗 Read the release blog and register for the live Q&A:
Show more
This is the first time I actually use this RTX 3060 for AI work. So far nothing seems to burn, temp at 81C Great GPU! It's nearly 4.5yo and I had no idea I would be quantizing LLMs with it when I bought it!
Show more
Statistically lossless quantization for LFM3 1.7B in ~48 hours! GGUF versions I'm using my very old RTX 3060 for this task!
Excellent LFM3 GGUFs from AtomicChat, as usual. These results also show how tricky it is to quantize small models while preserving BF16 fidelity. I usually aim for 98 or 99 EAR, but that looks tough for this model. 96 EAR might be a more realistic target. I’ll double-check the accuracy on a few task benchmarks. (Also interesting: the difference in next-token probability distributions between FP16 and BF16.)
Show more
This shows how hard it is to evaluate models using a single score. People tend to use models for just a few of the tasks these benchmarks cover, so they don't trust it when the overall score doesn’t reflect their experience.
Show more
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers. Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale. New to the Intelligence Index: ➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change ➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment Upgraded: ➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions ➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised ➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1 Key results: ➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50 ➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36) ➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
Show more
Statistically lossless quantization for LFM3 1.7B in ~48 hours! GGUF versions I'm using my very old RTX 3060 for this task!
Super impressive models! faster, better, and smaller than Qwen3.5 4B and 9B I'll try to find the time to make an SLQ version of LFM3-1.7B
Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence. > 6 model instances, across text, audio, vision, and omni modalities, generative and decision models. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding > 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages. > 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language. > 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input. > fine-tune on 1 GPU. > a new tri-hybrid architecture > architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1# on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒 Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device. 1/ 🧵
Show more
It's great to see custom inference engines becoming increasingly popular, but be careful about output quality. I ran some experiments last week and found how easy it is to get Astra or Opus to optimize an inference engine for a specific model or GPU, with impressive speedups. But LLMs can also take questionable shortcuts. Many of the speedups I observed came from much more aggressive arithmetic approximations. These approximations aren't necessarily bad, and output quality can remain excellent. Just make sure you validate it yourself.
Show more