🚀 Rapid-MLX 0.11.0 is out! Making local models on Apple Silicon reliable enough to run your agent workflows, not just demo them.
⚡️ Performance
• Prefix-cache reuse: 13.1s → 0.51s TTFT (25.6x faster on a repeated 6k-token prefix). Shared system prompts stop paying prefill twice. One session saved 38,032 tokens.
• Response cache: 656ms → 2ms (284x faster, byte-identical, zero GPU decode).
• Throughput: 152 tok/s on Qwen3.5-4B-4bit. Cold starts in seconds.
🤖 "rapid-mlx chat" grew into a real agent
Point it at any MCP server and it chains tools autonomously (e.g., filesystem server → 14 tools → list_directory → answer), complete with a live tool-activity UI. Thinking budgets force-close at decode time to keep things snappy.
🧠 5 New Model Families
• HY3 295B MoE (native MTP speculative decoding), Qwen3-Coder-Next 80B, MiniCPM5, LFM2.x, and a 2-bit Ternary Bonsai that actually runs.
• Kokoro TTS → Parakeet STT round-trip verified word-perfect.
• Qwen3-VL vision, KV cache export/import, and structured output on llguidance.
🛠️ Schema-valid tool calls by construction.
If you run coding agents on local models, you know the pain of "almost-JSON" arguments killing your parser. 0.11.0 constrains decoding itself—the model cannot emit a malformed tool call.
• Verified across harmony/gpt-oss, qwen3_coder_xml, gemma4, deepseek_v3, plus hermes.
• Auto, forced, and required. On by default, zero flags.
• The Keystone: It now holds for reasoning models too! tags no longer break forced calls because of grammar offset issues.
🛡️ Reliability & Honest Gaps
Every release runs an automated dogfood battery (5 models × 11 checks on real hardware). Invalid schema → honest 400. Unknown model → 404 with available options. No silent fallbacks. Gap: Qwen3.6 native MTP is parked for now, and a same-machine Ollama head-to-head is up next.
⬇️ Now in homebrew-core
brew install rapid-mlx
rapid-mlx chat qwen3.5-4b-4bit
Apache-2.0, MLX-native. 3.4k stars — break it and tell us, issues get answered fast. ⭐️
Show more
