Did you know you can now put the text of almost all of arXiv on one cheap SSD. Someone just packaged an enormous arXiv snapshot on Hugging Face 📚 3,148,796 papers 📄 PDFs for 99.47% 💽 Full archive: 16.08 TB Sounds ridiculous for Local AI… Until you see that … 🔥 paper_text = 70GB That contains the resolved TeX text for … 🧠 2,856,227 papers 📊 90.7% of all arXiv And you can download just that dataset by itself. No 16TB NAS is necessary. So theoretically you could build a completely local research system using ThumbLLM.exe from a thumb drive!!! 💾 70GB arXiv text corpus 🔎 local search / embeddings 🧠 local LLM 🚫 no API required Ask your local model to search nearly 3 million scientific papers sitting on your own SSD. 👀 Caveats ⚠️ It’s a snapshot, not live arXiv ⚠️ latest submission is Aug. 27 ⚠️ the text is resolved TeX, so it still contains LaTeX syntax/macros and needs some cleaning for ideal RAG But 2.8M papers in ~70GB is a Localmaxxer dataset. Link in ALT

K2 Horizon got another llama.cpp-family path, this time through TurboQuant. Spark-X2.5 already landed in upstream llama.cpp. I’ve actually got it running myself. A brand-new K2 Horizon TurboQuant PR adds … 🧠 K2 Horizon model support 📦 Hugging Face → GGUF conversion 💬 tokenizer + chat templates ⚙️ llama.cpp-style inference And they’ve already tested 🧠 K2-Horizon-3.7B Q4_K_M 🍎 M3 Pro / 18GB ⚡ ~35 tok/s 📚 64K configured context The PR also adds Spark-X2.5 to the TurboQuant fork: ⚡ ~36 tps 📚 256K configured context Caveats ⚠️ This is still an OPEN PR ⚠️ Only Metal has been tested ⚠️ Other K2 sizes + MoVA/MoE variants remain untested K2 Horizon already has an IFM llama.cpp fork, but it still isn’t supported in normal upstream llama.cpp. So what I’m watching here is whether TurboQuant becomes another path for K2.

How did I miss this?! Bonsai isn’t just an LLM family. Back in May, PrismML released Bonsai Image 4B, a crazy low-bit version of FLUX.2 Klein 4B designed to run locally on 🍎 iPhones. Look at this 🎨 FLUX.2 Klein 4B transformer 💾 FP16: 7.75GB 🌳 Ternary Bonsai: 1.21GB And the entire Apple deployment payload, including its compressed text encoder + VAE, is only: 🔥 3.88GB 📱 iPhone 17 Pro Max 🧠 A19 Pro / 12GB unified memory 🖼️ 512×512 ⚡ 9.4 sec/image 📲 MLX Swift ☁️ No cloud 💾 Bomsai Studio (App Store) M4 Pro: ~5.8 sec/image. No cloud. The ~4B diffusion transformer is paired with a 4-bit Qwen3-4B text encoder, which gets unloaded after the prompt is encoded to save memory. And there’s also 🍎 Mac / iPhone / iPad support 🟢 low-bit Gemlite builds for Nvidia GPUs 🔓 Apache 2.0 This is completely separate from the Bonsai 2 27B LLM I’ve been posting about. PrismML took a 7.75GB FLUX transformer and crunched it down to 1.21G and put image generation on an iPhone. 👀 And somehow I missed this for 4 months.

🤯 A 600B parameter model is going open-weight Oct. 15… …but only 27B parameters are active per token. 👈👀 Here is Step 5 Preview (this model is exciting) and here is why … Stats 👇 🧠 600B total parameters ⚡ 27B active/token 📚 1M context 👁️ vision + video 🤖 built for long-running agents 🔓 open weights Oct. 15 And that 27B-active number makes this really interesting for Local AI right? At roughly 4-bit, 600B parameters would theoretically be ~300GB of weights. Real-world quantization + overhead means I’d expect something more like ~320–350GB before accounting for KV cache and other runtime memory. So we’re potentially looking at 💾 ~384GB-class memory → Q4 territory 💾 ~256GB-class memory → aggressive Q3 territory And because this is MoE, only 27B parameters participate in each token’s compute. ⚠️ That does NOT mean this is a 27B model or that it’ll run in 27B-sized VRAM. The entire model still has to live somewhere. But with the right runtime, that could mean: Tiered memory !!!! 🎮 GPU → active compute 🧠 RAM → expert weights 💽 SSD → colder experts/offload And here’s the comparison that I really like Kimi K3: ~2.8T/ ~104B active Step 5: 600B / 27B active Yet both currently land at 44 on Artificial Analysis’ Intelligence Index. So Step 5 may deliver Kimi K3-class intelligence with roughly: 🔥 79% fewer total parameters 🔥 74% fewer active parameters This could make Step 5 a much more realistic monster model for local hardware. When the weights drop Oct. 15, my first question how small a machine can we get this thing running on? 👀

Qwen’s Image 2.1 released but it come with lots of improvements but just as many caveats. The old Qwen-Image-2512 was a 20B model with a massive: 💾 40.9GB BF16 image transformer But the new Qwen-Image-2.1 comes with … 🧠 7B visual generator 💾 14.2GB BF16 🔥 7.26GB INT8 already available for ComfyUI 👈👀 (Day-1) This little thing can … 🎨 generate AND edit images 🖼️ generate native 2K 🫥 create real transparent RGBA images 👥 use up to 10 reference images ✏️ preserve people/products while editing 🔤 render text 🎯 do masked/local edits Qwen basically took the huge local image model and reduced with a catch … ⚠️ The tiny 7.26GB image model comes with an entire pipeline. It uses a Qwen3-VL 8B encoder + VAE. ComfyUI already has a 6.31GB W4A8 encoder, and Qwen supports CPU offloading for smaller GPUs. Also… 🚫 Qwen-Image-2.1 is NON-COMMERCIAL under the new Qwen Research License. That’s especially notable because Qwen-Image-2512 was Apache 2.0.

On the SAME 48GB M5 Pro running 🧠 Qwen3.6-35B-A3B @ 210 tps and @ 143 tok/s at 32K context, the multi-agent handling is so much better than mere tokens. With 32K of context already cached, Splash produced the first token in ⚡ 123 ms vs 🐌 816 ms for oMLX And when Inco tested 16 simultaneous 32K requests with Qwen3.8-27B… Splash handled all 16. It’s built to handle heavy agent workloads with lots of context and repeated prompts even with multiple agents at once w/fast reuse of cached context

Check out this part of Athena. 1 Spark can be turned into a local multi-model agent server. Install both🧠 Qwen3.8 Flash-Next & 🧠 DeepSeek V4 Flash The Spark can't hold both in its 128GB memory at once, so Athena can checkpoint the current agent/session then .. 🧹 unload one giant model 🔄 load the other ⚡ switch in ~46 sec 🔌 keep the same OpenAI/Anthropic API endpoint Your client just talks to Athena. Both quantized models scored 91/100 on Athena's tool-use eval: 🛠️ 69 scenarios 🧰 52 possible tools 🔗 chained calls 🚫 knowing when NOT to call 🔄 error recovery 📋 schema-valid JSON ⚠️ Both models had trouble with one tool-output prompt-injection scenario. Athena is free for personal/research/education use, but the engine itself is currently proprietary/closed-source.

The more I look at setups like this, the more I think OCuLink changes the perspective on Local AI mini-PCs. That old RTX 2080 Super doesn't need to hold your giant LLM to be useful. Give the NVIDIA GPU the CUDA-friendly jobs like 👁️ vision models and 🎨 image generation and ... mini-PC's CPU/iGPU/system RAM handles the larger model or another workload when 8GB isn't enough.

📣 New Inferenceing Engine Alert! One DGX Spark is now running DeepSeek V4 Flash + Qwen3.8 Flash-Next at 262K context with ~1,000 tps PP. A new engine called Athena recently dropped for Nvidia GB10 systems. On a single 128GB DGX Spark you can get ... DeepSeek V4 Flash ⚡ 8K prefill: 1,126 tps 📚 256K: 948 tps 🚀 Decode @256K: 19.4 tps Qwen3.8 Flash-Next ⚡ 8K prefill: 1,071 tps 📚 256K: 961 tps 🚀 Decode @256K: 32.1 tps Going from 8K → 256K barely rocks on prefill performance. And Athena caches long conversations to disk. 👈👀 A 141,519-token conversation reportedly restores in: ⚡ 2.1 seconds vs ~2 min 20 sec to process it fresh. It also includes ✅ speculative decoding ✅ OpenAI + Anthropic-compatible API ✅ tool calling ✅ Qwen image/document input ✅ persistent agent context ✅ Docker install ✅ switch models without changing clients This is what I want from DGX Spark. 🎯 Huge models + huge context + usable speed on 1 box sitting on your desk. 👀 🔗 Link in ALT

Interesting HuggingFace post. What if the easiest way to make long-context Local AI faster is to simply STOP feeding it everything? A new HF forum experiment tested 600 distractor-heavy long-context prompts with Qwen2.5-7B 4-bit. Instead of sending the entire ~14K-token context, a MiniLM semantic selector kept only the most relevant ~60%. The result ... 📚 Input tokens 13,124 → 7,702 ➡️ -41% ⚡ Prefill 871 ms → 463 ms ➡️ -47% 🚀 End-to-end latency 944 ms → 519 ms ➡️ -45% 💾 Peak VRAM 11.71 GiB → 9.16 GiB And here's the weird part... 🎯 Token F1 actually IMPROVED: 0.534 → 0.581 So the model got: ✅ less context ✅ lower VRAM use ✅ almost 2× faster prefill ✅ lower total latency ✅ BETTER answers Why? Because more context isn't always better context. Coding agents and RAG systems often drag around enormous histories filled with irrelevant junk. ⚠️ This was a controlled distractor-heavy HotpotQA stress test, not proof that deleting 40% of every real-world context will improve accuracy. 🔗 Link in ALT.

Intel's 32GB Arc Pro B70 finally got a proper Qwen3.8-27B Local AI workout. So this is why the 32GB VRAM is important, even from an Intel GPU. Here is Gigazine's setup. Windows 11 using Unsloth Desktop + llama.cpp ... 🧠 Qwen3.8-27B Q4_K_XL 💾 30.0GB VRAM + ~1GB shared RAM 🚀 23.4 tps Then they tried Q8: 🧠 Qwen3.8-27B Q8_K_XL 💾 30.2GB VRAM + ~1.5GB shared RAM ⚡ 15.9 tps That's a full 27B-class model running at very usable speeds on an Intel GPU. And also tested text-to-image models. 🎨 Z-Image-Turbo 1024×1024 / 8 steps ➡️ ~5.3 sec/image after warmup 🎥 MiniMax H3 video also ran …but these are much slower, demonstrating where Nvidia's more mature AI software stack still matters. The hardware setup... 🎮 Arc Pro B70 💾 32GB GDDR6 ⚡ 608 GB/s bandwidth 🔌 230W And here's the interesting value angle. In Japan, the tested ASRock B70 was selling for: 💰 ¥298,054 while many RTX 5090s were selling above: 💰 ¥900,000 ⚠️ That's Japan-specific street pricing, NOT a universal B70-vs-5090 price comparison. But THIS is why the B70 interests me for Local AI. It won't beat a 5090 but it's giving you 32GB of VRAM at a much lower entry point. 23.4 tok/s on Qwen3.8-27B Q4 good enough for you? 🔗 GIGAZINE review in ALT

Okay, so specialized decision models are becoming the rage. Bespoke Labs just released Nimble, an open 9B decision model built from Qwen3.5-9B. And apparently they built it in ONE DAY. The recipe: 🧠 Qwen3.5-9B 🎯 LoRA fine-tune 📚 only 2,676 training examples ❌ no Jev distillation ❌ no RL ✅ open model ✅ open data ✅ open training recipe Instead of generating explanations or JSON, Nimble scores the allowed answers: YES / NO → A / B / C → route 1 / 2 / 3 → severity 1–5 Then returns the choice + probabilities. Example of Bespoke's 324-example held-out test: Qwen3.5-9B → 66.36% Qwen3.8-27B → 84.88% 🔥 Nimble-9B → 90.12% Jev → 93.21% And because it isn't generating a bunch of tokens: ⚡ ~106 ms median on H100 🍎 ~444 ms median on an M5 Pro 64GB 👈 👀 🎯 So this is the key, giant reasoning model doesn't need to answer every tiny agent decision. Let the big model think, then let a small local model handle the ... 🛠️ tool selection 🧭 routing ✅ verification 🚨 policy decisions 📊 scoring 🔄 retry / stop Fast. Local. No API call. ⚠️ Bespoke says this is a narrow synthetic 324-example evaluation, NOT a standardized System One benchmark. 🔗 GH: /bespokelabsai/nimble

Qwen3.8-27B at up to 144 tok/s on an M5 Max MacBook Pro?! You have to try this Splash Engine! It's a new open-source inference engine called Splash. Instead of being a universal runtime like llama.cpp or Ollama, Splash optimizes the whole stack around the exact model 👇 ⚙️ model-specific kernels 🧠 hardware-aware memory planning 🚀 DFlash2 speculative decoding 💾 prompt-cache reuse 👥 continuous batching On the same 48GB M5 Pro running Qwen3.8-27B: 🚀 Splash: 74 tok/s ⚡ oMLX: 38 tok/s 🐌 Ollama: 24 tok/s At 32K context: 🚀 Splash: 54 tok/s And with 4 concurrent requests: 🔥 170 aggregate tok/s vs 43 tok/s for oMLX Now Inco + LM Studio are showing up to ~144 tok/s on M5 Max. And LM Studio Bionic 1.1.5 already added Splash as an experimental runtime. Requirements 🍎 M3 or newer 💾 36GB minimum 👍 48GB+ recommended Completely different performance because the software stack is optimized around what it is running. ⚠️ 144 tok/s is an Inco/LM Studio result. The detailed numbers above are from Inco's 48GB M5 Pro test. 🔗

Nvidia dropped an official DeepSeek-V4.1-Flash NVFP4 build. And this thing is BIG. DeepSeek V4.1 Flash stats 🧠 552B backbone 📚 +196B Engram conditional memory ⚡ only 8B active during prefill 🚀 16B active during decode 👁️ native vision 📖 1 MILLION token context 🧩 384 routed experts across 40 layers 📜 MIT license Nvidia has converted its routed MoE experts to NVFP4 W4A4 specifically for Blackwell GPUs. This is not a compressed giant model down to 4-bit. DeepSeek's experts were already stored in MXFP4, Nvidia instead converts them to its Blackwell-friendly NVFP4 format. And because NVFP4 uses finer scaling, the checkpoint actually gets slightly larger. 💾 Source: ~476 GiB 💾 NVIDIA NVFP4: ~492 GiB 48 safetensor shards. 😳 So why bother? Because NVIDIA is optimizing how those 4-bit experts execute on Blackwell. And impressively, NVIDIA's evaluations show basically no obvious quality collapse from the conversion. For example: 🧠 GPQA Diamond 91.04 → 91.29 💻 SciCode 54.40 → 55.84 🛠️ Terminal-Bench 2.1 81.60 → 82.16 👁️ MMMU-Pro 74.05 → 73.70 Some slightly up. Some slightly down. Essentially benchmark parity. And it already has: ✅ vLLM support ✅ SGLang support ✅ reasoning parser ✅ tool calling ✅ image input ✅ 1M context ⚠️ Nvidia validated it on 4× GB300 GPUs so not a local model (yet). The checkpoint is still ~492 GiB. 🔗 HF: /nvidia/DeepSeek-V4.1-Flash-NVFP4

🤯 This is like having Jev at home, a AI reflex system running locally. Someone is turning Google's open DiffusionGemma 26B-A4B into a local version of the new “System One” AI idea inside vLLM. Remember Jev? It is a decision LLM. Instead of asking an LLM to generate paragraphs, System One models are designed to make FAST structured decisions: 🎯 yes / no 🧭 route A / B / C 🛠️ which tool to call 🚨 severity 1–4 📊 classify / score / choose And DiffusionGemma has an important role because a normal autoregressive LLM takes Prompt ↓ token ↓ answer DiffusionGemma is Prompt ↓ predefined answer slots ↓ fill multiple decisions in parallel Because it operates over a token canvas with bidirectional attention instead of being forced to generate everything strictly left-to-right. The vLLM patch lets it return ✅ bounded choices ✅ probabilities ✅ confidence / uncertainty ✅ multiple decisions simultaneously And it runs locally. On ONE DGX Spark the developer reports ⚡ 1 request: ~120 ms 🚀 concurrency 32: ~54 requests/sec 🧠 3 decisions/request 🔥 ~162 decisions/sec Then the developer found each question can require only around 3 canvas tokens: 🔢 index ❓ placeholder ✂️ separator Meaning as many as ~85 questions could fit into 1 diffusion canvas. And subsequent optimization reportedly cut decision time another: ⚡ ~20–40% with no change in decision quality. The model stats ... 🧠 25.2B total parameters ⚡ 3.8B active 💾 NVIDIA NVFP4: ~18.9GB 🎮 Can fit on a 24GB NVIDIA GPU 📜 Open weights ⚠️ vLLM PR #57250# is still OPEN 👉 not merged. And this is Jev-like functionality. It is NOT evidence that DiffusionGemma matches Jev's intelligence or calibration. 🔗 /vllm-project/vllm/pull/57250

Loving this OCuLink setup with the GMKtec EVO-T1 paired with an RTX 2080 Super. An Intel mini-PC gets dedicated Nvidia CUDA + 8GB of VRAM over PCIe 4.0 ×4 OCuLink. Local AI possibilities are endless! Could run separate models/workloads across the iGPU/CPU and NVIDIA GPU or experiment with model offloading, or even play with disaggregated inference!

🤯 Remember back in July I called Ling-3.0-flash one of the more interesting Local AI models? ... because Ling-3.0-flash combined 124B-scale capacity with only ~5B active parameters per token, strong frontierish benchmarks, and a design that looked promising for local agents. My July post ended with one BIG problem 👉 localmaxxers couldn't download the weights or load it into llama.cpp.” Well... that part changed. 🔥 @TheInclusionAI has now released the weights under MIT, an official GGUF exists for Ling-3.0-flash, and they've just released Ling-3.0-flash-VL. The new VL model adds: 👀 Images 🎥 Video 🤖 Visual/computer-use agents 🧠 124B total / ~5.5B active 📚 up to 1M context ⚖️ MIT A community Ling-3.0-flash-VL Q4_K_M GGUF is already ~79.3GB ~1.7GB vision projector. So roughly 81GB for a 124B multimodal MoE. That puts it in range of these devices ... 🧠 128GB Strix Halo 🍎 high-memory Apple Silicon 🎮 large/multi-GPU home rigs ⚠️ The VL GGUF is still experimental and currently uses a patched llama.cpp, so this is not yet a clean stock-llama.cpp experience. But this is exactly the update I wanted back in July, weights on our own machines. 🔥 Now somebody run that ~81GB VL stack on a 128GB Strix Halo and give us the tps. 👀 🔗 HF inclusionAI/Ling-3.0-flash-VL

🤯 Remember the weird U.S.-built diffusion LLM I posted about that broke 1,000 tok/s on Nvidia GPUs? It got a BIG upgrade. @_inception_ai released Mercury 2.5 today. 🎯 Instead of generating one token after another like a normal LLM, Mercury uses diffusion to generate/refine multiple tokens in parallel. And Inception says the new model delivers the following 👇 🚀 1,107 tok/s 👀 🧠 +40% intelligence vs Mercury 2 📚 260K context 🤔 Tunable reasoning 🔧 Parallel tool calling 📐 Structured JSON 💵 $0.20/M input / $0.75/M output 👀 🎯Inception says this is the largest diffusion language model ever trained. That ~1,100 tok/s isn't coming from Groq or Cerebras-style custom inference hardware - this is an important point! It's running on widely available NVIDIA GPU infrastructure. That's the big architectural bet, traditional LLM 👇 token → token → token → token Diffusion LLM 👇 many tokens → refine them together → answer ⚠️ Caveat, of course! 🔒 Mercury 2.5 is closed-weights.