Quantized Qwen3.8 27B for agentic coding: Same Score, 42 Different Outcomes.
I’ve been evaluating quantized Qwen3.8 27B models on DeepSWE 1.1, but I’m giving up.
4-bit AWQ run #
1#: 38.94% Reward
4-bit AWQ run #
2#: 31.86% Reward
32 of the 113 task outcomes flipped: 20 previously solved tasks failed, while 12 previously failed tasks passed.
On non-agentic tasks, I’ve found NVIDIA NVFP4 performs very closely to AWQ.
On DeepSWE 1.1?
NVIDIA NVFP4: 31.86% Reward
Exactly the same score as AWQ run #
2#, but 42 task outcomes flipped (nearly one-third of the benchmark!).
Each solved 36 tasks, with only 15 successes in common.
For reference, BF16 with the same Pi harness and thinking mode reached 43.36%.
Quantization certainly contributes to trajectory instability: small changes in token probabilities can alter an edit or tool call.
That changes what the agent sees next, potentially sending subsequent decisions down a different path. Errors can compound across turns.
Separating that from ordinary sampling variability requires repeated experiments. I’d want at least three independent full runs per model with fixed settings before drawing conclusions, and more for a reliable estimate of variability. Too long, too expensive.
Don’t trust anyone reporting a quantized model’s accuracy on agentic tasks without also reporting the run-to-run standard deviation.