Tested
@UnslothAI new Dynamic V3 GGUFs on a 4090. They deliver a Q3 that beats a Q4 4.5GB bigger.
All runs: same box, same harness, wikitext-2 (60 chunks), llama.cpp + DFlash2 speculative decoding, q4_0 KV, 262k ctx.
Quant | Size | PPL | Med t/s | Needle retrieval
๐Unsloth UD-Q3 12.24GB 6.3993 110.7 250k tok
IQ4_XS 15.1GB 6.4149 107.3 32k
Unsloth UD-Q4 16.7GB 6.4181 62.6 4k
My Q3 (imatrix) 12.57GB 6.5316 84.8 258k tok
My hand-tuned 13.1GB 6.5865 94.0 120k
Takeaways:
โข Their Q3 has the best PPL of anything I tested including Q4s
โข Smallest file, fastest median, and it retrieved an exact needle at 250k tokens
โข I tried hand-rolling my own layer mix (q6_K embeddings, q5_K attention). It was the worst result. The recipe is the value per-layer types derived from error analysis
โข Their Q2 does degrade (6.6469, +3.8%) โ Q3 is the floor where quality holds
Dynamic V3: