Here is the answer I was looking for Buried in the report Bonsai 2 27B's accuracy on terminal bench and swe bench drops by nearly 20 points. It's all I needed to know and what they should have published from the beginning in their main benchmark table.

Bonsai 2 has been evaluated with a low thinking budget for xhigh. Quantization errors really show their impact on long sequences, and Qwen3.8 27B often needs more than 81K tokens to complete its answer. For coding problems, like in LiveCodeBench, this is not enough. Expect some surprises for long-horizon agentic tasks. It's probably not as good as the model card says. Remarkable work nonetheless, as always.

This is what you get when you evaluate your quantization on easy benchmarks: community disappointment I really think Bonsai 2 is good, but the benchmark mix they used to compute this "98.2%", including oversaturated benchmarks like GSM8k and HumanEval, artificially inflates the accuracy recovery. It's nearly impossible for a recent model to score poorly on these benchmarks unless you break the model.

A proof that the benchmarks we are all using, are used to directly validate LLM training. In this case of Xiaomi live streaming, they can stop the training when the performance on deepswe, and all the benchmarks they target, is greater than some other baselines they have. This is not how benchmarks are supposed to be used. And that's why they are quickly "saturated" once released. It's obvious that most providers do something similar.

If you retry a lot, Qwen3.8 27B can reach 92.04% on DeepSWE 1.1, ~18 pts above GPT-6 Astra’s reported ~74%. I ran 20 configs across harnesses, thinking levels, and quantizations. Coverage varied a lot: two runs both scored 31.86% Reward, yet disagreed on 42/113 tasks. I saw this mainly for quantized models. I pooled results per task: if any single candidate passed all new tests, the task passed. No patch mixing. Best-of-k: k=1 → 81.43% F2P / 37.21% reward (that's the average single-run performance) k=10 → 98.31% / 86.25% k=20 → 99.18% / 92.04% This is oracle-selected, cross-config coverage, not single-run accuracy. It shows a 27B model can produce passing solutions for most tasks. The harder problem is reliability + selecting the right candidate. With strong independent acceptance + regression tests, retries to improve the result are less unrealistic: generate → verify → retry → stop on pass. Cost is also important: imagine if one Qwen attempt costs 1/20 of a frontier attempt, 20 retries ≈ 1 frontier attempt in generation cost. But that’s only a hypothesis until actual per-attempt cost, verification, infra, and latency are measured.

Quantized Qwen3.8 27B for agentic coding: Same Score, 42 Different Outcomes.  I’ve been evaluating quantized Qwen3.8 27B models on DeepSWE 1.1, but I’m giving up. 4-bit AWQ run #1#: 38.94% Reward 4-bit AWQ run #2#: 31.86% Reward 32 of the 113 task outcomes flipped: 20 previously solved tasks failed, while 12 previously failed tasks passed. On non-agentic tasks, I’ve found NVIDIA NVFP4 performs very closely to AWQ. On DeepSWE 1.1? NVIDIA NVFP4: 31.86% Reward Exactly the same score as AWQ run #2#, but 42 task outcomes flipped (nearly one-third of the benchmark!). Each solved 36 tasks, with only 15 successes in common. For reference, BF16 with the same Pi harness and thinking mode reached 43.36%. Quantization certainly contributes to trajectory instability: small changes in token probabilities can alter an edit or tool call. That changes what the agent sees next, potentially sending subsequent decisions down a different path. Errors can compound across turns. Separating that from ordinary sampling variability requires repeated experiments. I’d want at least three independent full runs per model with fixed settings before drawing conclusions, and more for a reliable estimate of variability. Too long, too expensive. Don’t trust anyone reporting a quantized model’s accuracy on agentic tasks without also reporting the run-to-run standard deviation.

Qwen3.8-27B NVFP4 variants are very close in accuracy. The real differences are memory and speed. If you're VRAM-limited, minima-ai/mnma_qwen3.8_27b_nvfp4 is a good pick, but it doesn't include MTP for faster inference. NVIDIA's version has MTP, but in my long-context coding tests MTP-4 is only ~2× faster than no MTP, and still ~2.5–3× slower than Unsloth (RTX Pro 6000). A likely reason: NVIDIA quantizes lm_head to NVFP4, while Unsloth keeps it FP8. Since MTP shares the target model's lm_head, this can hurt prediction quality and acceptance rate. So my pick is Unsloth NVFP4. Now testing accuracy on long-horizon agentic coding.

The more I think about it, the worse it gets. We’re not just evaluating a harness–model pair. We’re evaluating an adapter–harness–model triplet. The adapter is what you can easily benchmaxx. Take DeepSWE: Pi running through a basic Pi-to-Pier adapter would perform much worse than the same harness and model running through a carefully tuned adapter. Same model, same harness, different integration, potentially very different results. The adapter is rarely published. We really do have a benchmarking problem.

One of the most interesting tables in the DeepSeek V4.1 model card. Same model, same benchmark, wildly different scores depending on the agent harness. One more proof that comparing your new model’s score with previously published numbers is close to meaningless unless the harness and config are matched.

Qwen3.8 27B: mini swe agent vs Claude Code vs Pi What I learned With enough iterations, boosting scores on agentic coding benchmarks is relatively easy. On DeepSWE 1.1, Qwen3.8 scores poorly with vanilla Pi. But a benchmaxxed Pi setup beats (Reward) the Qwen team’s published result using Claude Code. Also: experiments with thinking low show that it spends more tokens, more turns, and score lower than medium. Full details, including an ablation study and token-efficiency analysis:

"Full" NVFP4 Qwen3.8 27B: Did you try this? I can't trust the eval. LiveCodeBench results for the BF16 baseline are 10~15 points below what they should be. Same for AIME. Evaluated with a low context length (32K), so in a condition where seeing quantization degradation is unlikely Just wondering whether I should spend one or two days on an RTX Pro 6000 to confirm, but at native context length.

I’m evaluating a batch of quantized Qwen3.8 27B models (non-GGUF) on 20K+ prompts across a variety of tasks: cyankiwi/Qwen3.8-27B-AWQ-INT4 — done unsloth/Qwen3.8-27B-NVFP4 — done EschaLabs/Qwen3.8-27B-Escha-W2 — done ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ — running Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw — pending (still figuring out whether it can support high concurrency; if not, it won't make it to the list) I can include one more model. Any recommendations (must be non-GGUF)? Maybe NVIDIA's NVFP4? I’m also evaluating all the models I made here: So far, everything looks good except for the smallest model, whose evaluation is still running. Some of my quantizations use 8-bit activations. While they don’t appear to hurt accuracy, I’m not seeing any significant speedup either (on Blackwell). I’ll run a few ablation experiments with 16-bit activations to get a better sense of the tradeoff. These are non-agentic evaluations. For agentic evals, I’ll focus on two models: the fastest 4-bit version and the lowest-bpw model that preserves accuracy. Once everything is done, I'll publish all the results, including latency and token efficiency.

If you don't know, Mistral AI's models are actually used a lot by the French tech ecosystem, even by small businesses Local media, and politics, treat them as untouchable, a true openAI competitor ,and French devs are convinced they’re using the absolute best AI on Earth ​Maybe that's also true in other parts of EU. In this context, $3B make sense I guess.