Speed on Bonsai 2 27b is interesting, especially in relation to the the MoE Qwen3.6-35b-a3b and its anticipated 3.8 version There is a size/speed tradeoff If you have 64 gb of memory for models, then it will probably be better to use the future Qwen 3.8 a3b because it will be faster, around 2x if it uses the same architecture But if you prioritize quality over speed TODAY, then Bonsai 2 27b is a better alternative because it is reportedly a high fidelity compression that would outperform a3b But if you have 32 to 16gb of memory, then rejoice! Your local model just got a huge bump in intelligence! Good day for local models 4 bit quants of a3b are around 20gb in weights, whereas Bonsai 2 27b around 6-7gb, depending on which bit packing you use Don't be fooled by 2 files in the repo. PTQ1_0 and PQ2_0 are the same model, just different ways to do bit packing If you use the smaller bit packing in memory, the PTQ1_0, you need to do more arithmetic operations. That is why prefill PTQ1_0 one is much smaller. But for decode, it's still the memory bytes that are the bottleneck, so the decode with PQ2_0 is lower Let me know if I'm missing something More details on my local profiling on the DGX Spark:

The vibeslopped UI will improve, but I can now say has finally caught up with present, after ingesting 6 months of data on open weight models here on X It now discovers and analyzes any new Hugging Face models that people post about within 1 day, and gives you theoretical max tok/s on your hardware, based on my recent formulation It also shows you what everyone is saying about all the models, be it positive or negative. Note that it is fully LLM-based and fully automated, so I might introduce a manual re-evaluation feature for people who would like to appear scores and interpretations. The scoring and all is all work in progress Here is LFM 2.5 2.6b which I am a fan of recently, for in-browser embedded use-cases Try to search other models, and let me know if it works/breaks for you!

I have a new job! Excited to announce that I will be working with Hugging Face to make local models work great in OpenClaw and other open agent harnesses! I will be building in public and documenting everything along the way, stay tuned!