oMLX 0.7.4.dev1 is out! With the new Prefill Accelerator, connect an M5 Max to your big Mac and prefill up to 2.9x faster! Prefill Accelerator (64K prompt, M3 Ultra 512GB + M5 Max 128GB) - DeepSeek V4.1 Flash: 1,060 -> 3,123 tok/s (+195%) - Qwen3.8 Flash-Next: 1,309 -> 3,666 tok/s (+180%) - GLM-5.3 Flash: 540 -> 1,388 tok/s (+157%) - MiniMax M3.1: 297 -> 569 tok/s (+92%) - An M5 Max 64GB still adds 30-118%, and decode speed stays the same. Faster DeepSeek V4.1 with A8 and buffer pool improvements - Prefill on M5 Ultra is 43-48% faster, 32K goes 2,631 -> 3,779 tok/s (+44%). - oQ4e builds now run INT8 Activation Prefill on M5. Faster Qwen3.8 Flash-Next - FP16 builds get the fused kernels. Decode on M3 Ultra goes 38.9 -> 63.0 tok/s (+62%), and +23% on M2 Max. - Decode after long prompts is 3-5% faster on M5 Max. Better for agents - Parallel tool results from Claude Code reach the model in call order, so Qwen no longer pairs a result with the wrong call. - Agent sessions on a cluster reuse the prompt cache across tool turns, 743 -> 9,266 cached tokens by the fifth turn. - Switch oMLX models inside Codex and Copilot with /model. Also in 0.7.4.dev1 - Ming-3.0 TTS with a 2x faster decode loop. - Ask AI can use an oMLX server on another Mac. - Check for model updates in the macOS app. - German across the web dashboard and the macOS app. - Batched decode with 6 requests on 8-bit models is 16% faster. A big thank you to everyone who sent PRs, reported bugs and tested builds for this release, and a warm welcome to the five new contributors!

oMLX 0.7.3 is out! Ask AI, the question bar for your local models in any app, now talks back. Press Option-Shift-Space, speak, and your model answers out loud with captions. It can also search the web, read links and look at your screen. Ask AI voice and tools - Voice runs on LFM3-Omni, LFM3-Audio and Qwen3-Omni. With LFM3-Omni 2B on an M3 Ultra, the first audio comes about 100 ms after you stop talking. - Cut in while it's speaking, or share your screen in the middle of a conversation. - Web search, link reading and screen capture work in both text and voice. - Prompt placeholders like {date}, {app} and {clipboard}. Faster DeepSeek V4.1 - INT8 Activation Prefill on M5 Ultra: 64K prefill 2,882 -> 3,507 tok/s (+22%), 195K 2,476 -> 3,200 tok/s (+29%). - 12 concurrent requests decode 56.3 -> 125.7 tok/s (+123%). - Engram prefetch adds another +15% to prefill at 128K. Many requests at once, much less memory - 8 Lightning MTP streams at 81K on Qwen3.8-Flash-Next: peak memory 248 -> 161 GiB (-35%), decode 180.7 -> 241.1 tok/s (+33%). - Batched decode on oQ models: 16 requests 146.9 -> 221.9 tok/s (+51%) on M3 Ultra, 8 requests +74% on M2 Max. - Hybrid Qwen models reuse a shared system prompt across conversations, TTFT 0.89 -> 0.25 s. Also in 0.7.3 - Update downloaded models by fetching only the files that changed. - Better oQ quants, KL drops 9-49% for under 5% more size. - LFM3 text, audio and d1 models, and Unlimited-OCR-2. - Temporary chats in the web UI. Thanks to everyone who sent PRs and tested the rc!

oMLX by @jundotkim is now available as an engine in LM Studio and Bionic! Jun is brilliant and diligent and he's been doing amazing work with the oMLX project. We're super excited to partner up, and now ship this performant engine for your MLX models on your Mac.

oMLX is now available in LM Studio! 🎉 Get oMLX’s SSD KV cache, which keeps long contexts ready across sessions, and run the latest open frontier models on one platform. Thanks @lmstudio and @yagilb for making this happen!

MiniMax M3.1 Flash runs on your Mac with oMLX 0.7.3.dev1! With its bundled DSpark draft running as Lightning MTP, one request on an M3 Ultra (oQ4e) goes from 22.5 to 42.0 tok/s (+87%), New fused decode kernels then add another 6-7% with Lightning MTP! Prefill runs at 370 tok/s at 4K and still 298 tok/s at 128K. oQ builds on mlx-community - oQ3e (196GB): - oQ3.5e (213GB): - oQ4e (249GB): Also in 0.7.3.dev1 - Faster M1 and M5 Macs. Qwen3.6-35B-A3B prefill on M1 Ultra goes 429 -> 644 tok/s (+50%) at 100K, and DeepSeek V4 Flash prefill on M5 Max 650 -> 767 tok/s (+18%) at 32K. - Faster DFlash on fp16 models. Qwen3.8-27B fp16 + DFlash2 on M3 Ultra goes 106 -> 141 tok/s (+33%). - Faster, leaner GLM-5.3 Flash. Two requests at 64K decode 16.4 -> 19.2 tok/s each (+17%), and the model uses 3.27 GB less memory. - Better low-bit oQ. More of the bit budget now goes to the layers every token uses, not the output head. Low-bit quants come out closer to the original model. - More models. Flint, Raon-1 mini, and MiniCPM-V 4.7 with its DSpark draft.

It's only been a day since 0.7.1, but oMLX 0.7.2 is out with changes worth shipping now. This release brings faster DeepSeek V4.1 on every oQ build, Lightning MTP in distributed inference, and Qwen3.8-Flash-Next GGUFs on Splash 1.4.0. Faster DeepSeek V4.1 on oQ2e to oQ4e - oQ3e with Lightning MTP + CED: decode 49.4 -> 68.2 tok/s (+38%), prefill 700 -> 949 tok/s (+36%). Every quant from oQ2e to oQ4e gets faster, with similar prefill gains. - Prefill holds around 900 tok/s all the way to a 128K prompt. - Measured on M3 Ultra 512GB. I'll run more tests on M5 Ultra 256GB. Lightning MTP in Distributed Inference - DeepSeek V4.1 and GLM-5.3 run Lightning MTP across Macs with both PP and TP. - With TP on two M3 Ultras, both models decode more than 50% faster, even over a TCP ring. I'll share exact numbers after more tests over RDMA. - A pipeline stall is fixed, so DeepSeek V4.1 across two Macs went from about 1 tok/s to about 28 tok/s. More Models on Splash - Qwen3.8-Flash-Next GGUFs run on Splash 1.4.0, and oMLX offers the matching DFlash2 draft. - Split GGUFs download and show up as one model, and mlx-community OptiQ builds now start. Other Improvements - Better prompt cache reuse in Claude Code sessions, on both the built-in engines and Splash. - Reasoning models answer Claude Code's auto-mode classifier before it times out. - DFlash works with ParoQuant Qwen models. - SpecPrefill no longer disturbs other requests decoding next to it, and Gemma 3 gets the right positions in sparse prefill. - Keyboard access to the web UI menus and better contrast in the dark theme. Thanks to everyone who sent PRs and tested 0.7.1 so quickly!

oMLX 0.7.1 is out! This is the official 0.7.1 release. It adds GGUF models through Splash, much faster DeepSeek V4.1, 4-bit / 8-bit KV cache formats, and distributed inference for GLM-5.3 and DeepSeek V4.1. What's new - GGUF models through Splash. The downloader filters repos by MLX or GGUF and lets you pick the files, and Qwen3.8-27B and Qwen3.6-35B-A3B GGUFs run on Splash. - Affine4 and Affine8 KV cache. Full-attention KV is 74% smaller with Affine4 at 32K context. - A fix for GPU hangs when several models run at once. - Distributed inference for more models, GLM-5.3 and DeepSeek V4.1 now run across Macs with both pipeline (PP) and tensor (TP) parallelism. - Claude Desktop support. One toggle maps claude-opus, claude-sonnet and claude-haiku to your local models. - Realtime mic transcription with every STT model. Speedups in this release - DeepSeek V4.1 Flash decode on M3 Ultra: 20.9 -> 34.3 tok/s (+64%) for one request, 19.4 -> 91.8 tok/s (+373%) with 8 concurrent requests. - Qwen3.8-Flash-Next with Lightning MTP on M5 Max: decode holds 87-98 tok/s instead of swinging between 64 and 93. - Qwen3.8-Flash-Next with expert offload on M1 Max: +5-8% decode. Coming from 0.7.0? You also get Muse Spark Open, the Recommended models tab, DFlash in the batched engines, decision models, batched prefill and the improved oQe quantization. Thanks to everyone who sent PRs and tested the dev and rc builds!

Try Muse Spark Open right now in oMLX 0.7.1rc1, out today! Muse Spark Open is Meta's 816B MoE with 42B active parameters. In oMLX it reads text and images, calls tools, and lets you pick the reasoning level from low to xhigh. - Ready to download: This is the release candidate for 0.7.1, so it also brings everything from the 0.7.1 dev releases - Splash as an external engine for Qwen3.8 and Qwen3.6 - A Recommended tab that shows which models fit your Mac - Reworked model settings and chat in the web UI and the macOS app - DFlash inside the batched engines, with continuous batching and the prefix cache - Decision models (Clef, OpenJev) and batched prefill for concurrent requests

oMLX 0.7.1.dev2 is out! oMLX can now serve Qwen3.8 and Qwen3.6 through Splash as an external engine. Splash runs the model on its own Metal engine with DFlash2, and the model still shows up in oMLX's model list, API, dashboard, cache stats and benchmarks like any other. This release also helps first-time users find a model that fits their Mac without guesswork, reworks model settings and chat in the web UI and the macOS app, and speeds up GLM-5.3 and DFlash. Splash as an external engine - Turn it on per model in Model Settings > Advanced. oMLX finds the Splash launcher and hands it the model and DFlash2 draft from your model directories, so nothing extra is downloaded. - Memory and context window are fitted to your oMLX memory limit, and an MLX model can run alongside it. - Vision checkpoints take images too. Speedups in this release - GLM-5.3-Flash decode on M5 Ultra: 85.5 -> 95.3 tok/s (+11%) at 64K context. - GLM-5.3-Flash, 4 streams on a shared 32K prompt on M5 Ultra: 81 -> 91 tok/s (+12%). - DFlash with 4 concurrent requests on M3 Ultra: +82% to +157% (MiMo V2.6 Flash, Qwen3-4B, Muse Glimmer 30B). What's new - Find the right model for your Mac. New users no longer have to guess which model to start with. The new Recommended tab shows only oQ models that will actually load on your Mac, with Lightning MTP on by default. - Reworked model settings. The web UI groups them into sections with a side menu, and the macOS app splits them into pages that save as you edit. - A new chat composer. Pick the model, profile and reasoning level right where you type, with a reasoning slider that follows the levels each model accepts. - DFlash now runs inside the batched engines, so it gets continuous batching and the prefix cache. DSpark drafters are accepted 61-85% of the time instead of 16-21%. - Requests survive memory pressure. They are paused and resumed instead of aborted, and cancelled prompts stay cached. - /v1/decisions for any chat model, plus the Decision 3.0 models with image and video input. - YaRN for Qwen3.5-family models past the native 262K context. - An "Allow inference without API key" toggle for LAN setups, and a Mistral Vibe launcher. - oQ for DeepSeek V4.1 at oQ2 to oQ2.7 and for EmbeddingGemma 2, plus MiniCPM-V 4.7 support. - A log table in the macOS app, copy buttons for model names, and Downloaded badges in the model browser. Most of the speedups came from contributors again. Thanks to everyone who sent PRs and tested the dev builds!

Fun addition to transformers serve: you can now turn any of the 250 LLM & VLM model into a JEV-like decision model. /v1/systemone scores a set of decision with the model's logits. No need to generate text! Which one of them can get microducks across the finish line fast?

All my oQe models have been updated with Improved oQe Quantization, available from oMLX 0.7.1. Compared with the previous oQe builds, KL divergence is 11.6–18.1% lower on code and 10.2–13.8% lower on prose, so please give them a try. The full model list is below. KL divergence from the original BF16 model (mean, lower is better) - Qwen3.8-27B oQ4e: code 0.0143 → 0.0117 (-18.1%), prose 0.0696 → 0.0600 (-13.8%) - Qwen3.8-Flash-Next oQ4e: code 0.0877 → 0.0776 (-11.6%), prose 0.1909 → 0.1713 (-10.2%) What's new - Below oQ4, the Engram and N-gram tables stay at 4 bits and are left out of the bit budget. oMLX can serve these tables from SSD instead of memory, so the full bit budget goes to the rest of the model. - Model cards now show the bit distribution and storage breakdown. If you downloaded an earlier oQe build, please re-download it. Improved oQe models (69) - Qwen3.8-Flash-Next: 2.5 · 2.7 · 3 · 3.5 · 4 · 6 · 8 bits - Qwen3.8-Flash-Next FP16: 2.5 · 3 · 3.5 · 4 bits - Qwen3.8-27B: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - Qwen3.6-35B-A3B: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - GLM-5.3-Flash: 2 · 2.5 · 3 · 3.5 · 4 · 5 · 6 bits - DeepSeek-V4.1-Flash: 2 · 2.5 · 3 · 4 bits - DeepSeek-V4-Flash-0731: 2.5 · 2.7 · 3 · 3.5 · 4 bits - gemma-4-31B-it: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - gemma-4-26B-A4B-it: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - gemma-4-12B-it: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - clef: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits - clef-flash: 2.5 · 3 · 3.5 · 4 · 6 · 8 bits

oMLX 0.7.1.dev1 is out! This dev release adds decision models (Clef, OpenJev), batched prefill for concurrent requests, a refreshed web UI, and faster Qwen3.8, Qwen3.6, GLM-5.3-Flash and DeepSeek V4.1. Speedups in this release - Qwen3.6-35B-A3B decode on M5 Max: 156 -> 205 tok/s (+31%). - GLM-5.3-Flash decode on M3 Ultra: 31.3 -> 40.6 tok/s (+30%). The fused decode kernels now run on M3 and M4, not only M5. - Qwen3.8-Flash-Next with expert offload, 32K prefill on M3 Ultra: 396 -> 636 tok/s (+61%). - Qwen3.8-Flash-Next, 8 prompts arriving 0.4 s apart on M5 Max: time to first token 7.41 -> 5.75 s (-22%) with batched prefill. What's new - Decision models. Clef, Clef-Flash and OpenJev answer typed questions with a probability for each option through the new /v1/systemone endpoint. - Batched prefill. Prompts that prefill at the same time now share one forward pass, so MoE models read their expert weights once. - A refreshed web UI with a log table, a settings rail and a customizable dashboard. - Native video input for Qwen3.5, 3.6 and 3.8. - More accurate oQe quantization, with 11-15% lower KL to the original model. - Fewer memory guard rejections in long sessions. Most of the speedups came from contributors again. Thanks to everyone who sent PRs and tested the dev builds!

Please stop using MLX 4bit affine quants, focused on something else. They are fast but not good enough, especially in larger contexts. I learned the lesson the hard way using them and even thanks to @ggerganov pointing me in the right direction a long time ago.

After a long wait, oMLX 0.7.0 is out! This release brings faster Qwen3.8, GLM-5.3-Flash and MiMo V2, faster Lightning MTP and DFlash2 at long contexts, and a completely rebuilt memory guard. * Special thanks to @Spangler3000 and @jerry543, who helped a lot with the speed improvements. Performance vs 0.7.0rc1 (oQ4e, Lightning MTP on) - Qwen3.8-Flash-Next prefill on M5 Max: 1,953 -> 2,844 tok/s (+46%) at 16K context. - Qwen3.8-27B decode on M3 Ultra: 46.9 -> 70.3 tok/s (+50%) at 64K context. - GLM-5.3-Flash prefill on M3 Ultra: 413 -> 516 tok/s (+25%) at 16K context. What's new - A completely rebuilt memory guard. It lets oMLX use as much of your memory as it safely can. - DFlash2 support for GLM-5.3-Flash. - Faster image follow-ups for Qwen3.8-Flash-Next with a vision feature cache. - Fixes for the regressions reported against the RC, including ANE prefill, M5 A8 prefill and clustering. Most of the speedups in this release came from contributors. Thanks to everyone who sent PRs and tested dev&RC versions!

MLX now has fast VJP implementations for attention and linear attention, which utilize neutral accelerators. They make training/fine-tuning much faster on Macs, along with massive reduction of memory uses.