As much as I'd like this to work, 1-bit / ternary models like Bonsai are (today) a dud.
The sweet spot appears to be ~4 to 5 bits per parameter. That's what
@UnslothAI find to be the "knee" of the pareto curve (thanks
@realGeorgeHotz for pointing this out).
Interestingly a synapse in the human brain uses about 4.7 bits of information so perhaps this points to something more fundamental.
You're much better off using a smaller model quantized to 4-bit than a larger model quantized to 1-bit or ternary.
We also find these 1-bit / ternary models to be overly verbose - so they can actually run *slower* on end to end tasks than 4-bit, even though the decode TPS is faster.
98.2% of Qwen3.8 27B my ass.
I got hyped and gave it a real agent job right away. Build me an FPS in three.js, 6 hours on a 3090. My most standard and default prompt that I always use. It spent the first 32K tokens on a plan without writing a single file, then shipped a black screen, 2 shaders that dont compile and a player who spawns dead. And wrote "verified" in the final report.
Ok, too hard. I gave it the easiest thing I have, a voxel pagoda garden in one html file. Video attached. 3 hours for THIS. On the way it deleted its own file and spent an hour debugging a raycaster nobody asked for.
Where the 98.2% is in all this I have no fucking idea. Same Qwen3.8 27B, same pagoda task, ISTA-DASLab GSQ-RCO IQ2_XS on a 12 GB 3080 Ti gave me day and night, real shadows and koi fish in the pond. 8.4 GB on disk, real 2.50 bpw, 131072 ctx with q4_0 KV, 47 tok/s at 128K. 2.5 bits beats 2.13 bits by a lot when the 2.13 is this shit. Post with the video and the exact command in the replies.
In the replies, I'll attach what a proper Qwen3.8 27B created in my hands. Bottom line: if you have at least 12GB VRAM, use ISTA-DASLab GSQ-RCO of all THIS.
Show more