Google dropped MTP versions of Gemma4. Ran them on my DGX Spark. The 31B dense model went from 3.94 → 8.91 tok/s. That's +126%. Full results: [26B A4B] > 25.24 → 31.69 tok/s (+25.6%) > TTFT 755 → 332ms (-56%) [31B] > 3.94 → 8.91 tok/s (+126%) > TTFT 599 → 378ms (-37%) If you're not running MTP, you're leaving free perf on the table.

This might be my new favorite all-purpose local AI model. Fits nicely into 16GB of VRAM for an RTX 5060/5070TI. The new SparkX2.6-4B beats Qwen3.5-9B in 80% of shared benchmarks and 97% of Gemma4-12Bs. Stats 💾 official Q4_K_M → 3.54GB 🧠 only 4B parameters 📚 native 1M context 💭 none / low / high thinking budgets 🤖 trained for local agents 🔓 Apache 2.0 It already runs with: ⚙️ llama.cpp 🦙 Ollama 🖥️ LM Studio 🍎 MLX 🚀 vLLM / SGLang In the team's reported benchmark suite Averages: Spark-X2.6-4B → 63.3 Spark-X2.5-4B → 52.1 Qwen3.5-9B → 49.1 So they're claiming a 4B model beats their tested 9B comparison overall. For normal local-agent contexts, though, a ~3.5GB reasoning/agent model with this runtime support is very interesting. Link in ALT

We're working on making the local model experience better in Hermes, what are the best local models at each weight class? My blind guess, please correct: 8-16 GB VRAM Gemma4 12B 24-32 GB VRAM Qwen3.6 27B Qwen3.6 35B 128 GB VRAM (Spark, M3 Max) ??? Can you do DSv4-Flash?

Looking through key vLLM tickets. Probably time to update vLLM. 👇 v0.31.0 brings some practical improvements for local AI servers ⚡ Restart without reloading the weights. vllm preload keeps model weights in GPU memory across engine restarts. Useful when tuning your serving setup. This also supports MTP draft models. 🚀 More speculative decoding options. Model Runner V2 adds draft-model support, alongside async DFlash and DSpark adaptive verification for Gemma4. Benefits depend on your model and configuration. 🛠️ Fixes for stalled generation. The release addresses a KV-cache/MTP deadlock and HiSparse issues involving speculative-token acceptance, prompt processing and caching. 🎛️ Better control when requests pile up. New scheduling controls manage active requests and adapt how long prompts are processed. Link in ALT

717 commits. 307 contributors. 96 first-timers. vLLM v0.31.0 is live. 🎉 Highlights: 🤖 DeepSeek-V4.1-Flash: FlashMLA mega attention with NVFP4 KV caching, DeepGEMM sparse MQA, Mega-Gate, and shared Engram host tables ⚡ Fast restart: vllm preload keeps weights in GPU memory across restarts, with data parallelism and MTP draft support; experimental engine snapshots 🛠️ Model Runner V2: draft-model speculative decoding, LiLiCorr, async DFlash, and DSpark adaptive verification for Gemma4 🌐 Large-scale serving: MoonEP balanced all-to-all, prefill context parallelism with data parallelism, DeepEPv2, and RL weight transfer 🎛️ Smarter scheduling: independent active-sequence limits, adaptive long-prefill chunking, and priority for requests already holding KV blocks 🗄️ HiSparse hardening: fixes for MTP verification, chunked-prefill livelocks, and host/GPU prefix caching Thanks to everyone who made this release possible! 💙 Full release notes 👇

🚀 ZDTaichu5.0-9B is now on ModelScope! 🤖 An on-device multimodal model from TaichuAI. At 9B parameters it runs on a single GPU and brings spatial reasoning, embodied AI and agentic tool use to edge deployment. Qwen3.5-9B backbone + C-RADIOv4-H vision encoder, 128K context, any-resolution image and video input. 🧭 Spatial reasoning: leads the compared 10B-scale open VLMs (Qwen3.5-9B, STEP3-VL-10B, gemma4-8B-E4B) and scores above Gemini 3 Pro, Grok 4 and GPT-5.2 on ViewSpatial, MMSI-Bench and MindCube-tiny 🛠️ Agent: highest among the compared open models on TAU2-Bench, Claw-Eval and IFEval 📄 First-tier results on documents, charts, OCR, visual math and video, with a ready-to-use vLLM branch and Docker image 🧠 Entropy-Gated Adaptive Recurrent Reasoning: extra latent refinement steps go only to the hard tokens

Gemma is only as powerful as the community building with it. We launched awesome-gemma to highlight amazing community-made tools, research, and applications in the ecosystem. What have you built? Drop a link below or submit a PR to get featured: