i came across this llama.cpp fork that uses less VRAM and KV memory. on my GX10, Qwen 3.5-family Bonsai 27B, 60k context, median of three runs: q8_0: • 2,006 MiB KV • 792 tok/s prefill • 27.2 tok/s decode KVarN5 + 1,024-token tail: • 1,349 MiB KV • 740 tok/s prefill • 25.6 tok/s decode The trade: 32.8% less KV memory for roughly 6% lower throughput might be a solid option for people trying to squeeze in some more tok/s ! repo:

This is where we are right now. And i’m not gonna lie it feels pretty magical 🧚‍♀️ Qwen3.6 27B running inside of Pi coding agent via Llama.cpp on the MacBook Pro For non-trivial tasks on the @huggingface codebases, this feels very, very close to hitting the latest Opus in Claude Code, or whatever shiny monopolistic closed source API of the day is. In full airplane mode. Most people haven’t realized this yet. If you have, it means you have a huge headstart to what I call the second revolution of AI. Powerful local models for efficiency, security, privacy, sovereignty 🔥

Updates since then: * Deepseek v4 is out. There *is* a 2-bit quant that can run within 90 GB ( ), and it works, however it's only fast on Apple hardware (I've head ~35 tok/s). On AMD, it's ~7 tok/s. IMO actually taking the effort to properly support more than one hardware manufacturer is a great example of the difference between mere "decentralized AI" and genuine "CROPS AI". I hope we can become better at this. * also has alpha telegram support now. However, the path to adding your account is quite janky * looks promising as a way to run "dense" models (eg. Qwen 27B) more efficiently. It's janky, but on my 5090 laptop it seems to be ~2x more tok/s than llama.cpp * VoxTerm (local AI recording, no third-party servers) continues to be developed And there's a lot more projects coming on the horizon. One other thing that has been on my mind is that there's actually a lot of intersection between "CROPS ethereum access layer" and "CROPS AI". For example, we want a ZK way to make (paid) calls to remote LLMs. But if we have this, then it's just as useful for solving another problem: private RPC reads in Ethereum. Another example: application-specific finetuned LLMs. Leanstral ( ; I get ~38 tok/s on AMD) fits into < 70 GB, but can hold its own against 1T models on writing Lean code. Things like this are a huge boon for writing more secure code ( ). We should have models finetuned for Ethereum-related use cases as well.

Local AI is having its moment! Below is the number of new GGUF models created each month over the past 8 months & insights from our HF internal agent (May is partial): - 176,000 total public GGUF models on HF - Two distinct regimes: Oct–Feb averaged ~5.1K new GGUF models/month. Then March–April jumped to ~9.2K/month — nearly double the previous rate. - March was the inflection point (+55% MoM) — likely driven by a wave of new open-weight model releases being quantized to GGUF. - April sustained the momentum at 9.7K, suggesting this isn't a one-off spike but a new baseline. - The GGUF ecosystem is accelerating — the community is quantizing models faster than ever, likely thanks to better tooling (llama.cpp improvements, automated quantization pipelines, and more models supporting GGUF natively). Let's go!