Running a 763B model at 192K context usually demands a multi-node cluster. Meet Abyss — running DeepSeek-V4.1-Flash locally on a single dual-GPU workstation (2× RTX PRO 6000 96GB) by intelligently tiering native MXFP4 experts across VRAM, RAM, and NVMe. Latest benchmarks: ⚡ ~52 tok/s decode at 96K context 📖 ~14s prefill for 48K tokens (~3,400+ tok/s) In earlier matched tests against upstream llama.cpp, Abyss already delivered 5.2× faster prefill and 25% higher decode throughput—and our latest kernel & tiering optimizations push it even further. Code & details: #LocalLLM# #DeepSeek# #AIInfra# #CUDA# #Blackwell# #OpenSource#

Your local LLM can now take control of a browser and complete tasks autonomously via Unsloth! Qwen3.8-27B successfully controlled the browser to complete tasks like chess, painting, with no human intervention. GitHub: Guide:

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers don’t mean it’s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: ↳ When you download a model, it compiles that model’s kernels locally. ↳ It tries different kernel configurations and keeps whichever runs fastest on your chip. ↳ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: (don’t forget to star it 🌟) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don’t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.

Choosing the right local LLM tool can be tough. @cedricclyburn shares his take with a tier list to make the decision a little easier. How would you rank them? Let us know below.

🚨 Host Your Own Local LLM On The Abacus AI SuperComputer Stop being sad about Fable and control your destiny by hosting your own LLM - host open source LLMs like Qwen and Gemma - create chat bots or always on APIs - message it via an always on agents Use SOTA models like GPT 5.5xHigh or Opus to create open-source LLM apps and games

🚨 Host Your Own Local LLM On The Abacus AI SuperComputer Stop being sad about Fable and control your destiny by hosting your own LLM - host open source LLMs like Qwen and Gemma - create chat bots or always on APIs - message it via an always on agents Use SOTA models like GPT 5.5xHigh or Opus to create open-source LLM apps and games

Wrapped zkAPI + Tor + extra cover traffic + local LLM prompt rewriting in an open-AI compatible daemon, and made a "paranoid mode" wrapper for my agent harness (pi): Each conversation gets assigned a separate Tor circuit, your requests are masked with random cover traffic, zkAPI ensures you don't leak your identity through payments, and all prompts are locally rewritten (bring-your-own-local-model; I'm using qwen3.8-flash-next) My agent harness is wrapped with a bwrap sandbox (also experimenting with MicroVM) and wrapped again with torsocks, so everything it does goes through Tor. Here's what me asking it to do a simple research task looks like. It's getting quite usable! Though you definitely still pay quite a bit in increased latency, especially when starting a new conversation. Also, Tor is far from optimal here for privacy; the cover traffic mechanism is okay against passive adversaries but much weaker if the LLM provider is colluding with the guard and actively manipulating timing. This kind of task really requires a more specialized mixnet, probably take the tradeoff of a higher bandwidth multiplier and built-in constant cover traffic. Also also, much cheaper zkAPI deposit and withdraw gas costs are coming to mainnet soon.

Learn about how to get multipliers in local LLM speed-ups using speculative decoding in this livestream with @mervenoyann, @MaartenGr and @_EldarKurtic

Wild paper from Microsoft and colleagues. They show a new attack that reconstructs the text a local LLM generates by watching CPU cache activity while it detokenizes. Earlier cache attacks needed something unusual in the deployment, such as shared data memory, CPU offloading, or a Mixture-of-Experts architecture. This work targets the detokenizer, which runs in default inference pipelines. The method has two stages. 1) Flush+Reload on shared tokenizer code detects when decoding happens, which lets the attacker fire Prime+Probe at the right moment and isolate token-dependent cache activity. 2) A clustering and language-model pipeline then recovers readable text from the noisy observations. They evaluate across datasets, hardware platforms, inference frameworks and model families, including real local deployments and agentic systems. The widely used tokenizer implementations are susceptible, and they are embedded in many popular local LLM products and agent frameworks. OpenClaw is demonstrated directly. Paper: