๐ Holy ๐ฉ! Major Local AI Breakthrough! ๐ง 744B-parameter GLM-5.2 (1.5 TB total) is now running on just ~25 GB RAM โ no discrete GPU required! ๐
Wut!? Must test!!
Italian engineer
@JustVugg built Colibrรฌ, a pure C inference engine (single ~2.4k line file, zero runtime deps) that:
โข ๐ก๏ธ Keeps the dense core (~10 GB at int4) resident in RAM
โข ๐ Streams 21,504+ MoE experts from fast NVMe on demand (only ~40B active per token)
โข โก Supports native MTP speculative decoding + MLA attention
Result:
Frontier-class model on everyday consumer hardware!
๐ Current speeds:
โข 25 GB RAM setup โ 0.05โ0.1 tok/s (disk-bound)
โข Higher RAM + fast SSD โ up to 1+ tok/s (warm)
โข It's a start... what could you do with a 5090?
๐ก Big opportunity: Pair it with Phison aiDAPTIV+ AI SSDs to kill the I/O bottleneck ๐ smarter caching, prefetching & KV offload could make it dramatically faster!
This is a huge step toward truly accessible local frontier AI.
๐ GitHub: JustVugg/colibri