David Dalcu
@ddalcu
I build things, local LLM engines & UI’s. Created for Desktop and MLX Chat for iOS. (Zig based engine)
583 Following 878 Followers
This was peak Mac OS… needs to go back to being fast and lightweight, nobody wants 2000 background processes you can’t stop, eating up your ram.
Happy 17th Birthday to the greatest Operating System of ALL TIME:
OS X Snow Leopard! 🥳🎂🍾
Me trying to keep up with all the open-weight model releases and test them all
Does anyone want to attempt to squeeze more performance from a M5 Max & Qwen Flash Next ?
Here is the plan:
Please submit a PR, and I will code review it and merge it.
I don't have a M5Max, otherwise I would do it obviously... I heard @ivanfioravanti likes challenges like this 😅
Show more
@ddalcu Did exactly the same test as you but with Macbook Pro M5 max. Really cool, now I will connect it to Hermes and try some more. Thanks for fantastic work!
Awesome article from @aishwa_rya about using MLX-Serve to solve a real business need in the wild.
She switched to Gemma 26B A4B, the same model that is currently in the challenge from @yukonresearch because she has a base M5, speed was important to her. It's about to get a lot faster !
Show more
I jumped on the open weights hype and moved my workflow from GPT 5.4 to Qwen. Learnt I had the wrong model for the job, and got my AI bill to $0 in the process.
Detailed learnings here:
Show more
The MSRP of the new M5 Ultra Studios from Apple have completely obliterated the value economics for most consumer AI systems when measured as:
(model hosting memory)x(memory speed)/(cost of system)
Memory size and speed per dollar is off the charts (2-3x+ the current economics of the DGX Spark and Strix Halo) and I’d feel very confident assuming these 1.5x from MSRP very quickly.
The other systems won’t get cheaper so the only path for these studios is to price up into their market value. Much like the M5 laptops did.
There’s PLENTY to debate on what hardware works for you. But the economics of this initial offering are quite wonderfully broken ATM.
Show more
Qwen 3.8 Flash-Next is alive on MLX-Serve, under 128gb, working on performance next. MTP gets ~20tok a second, no MTP gets 30+ ... investigating why..
Qwen Flash Next is out, and not really 3.8, it's 4.0.. trying to add support for it now in MLX-Serve, will have a preview build later today. Im on Starlink, so download/upload is a bit slow.
Expected ram needed @ 4bit: ~80gb.
Show more
Here is your Mac Studio justification
Facts
A fun compare:
- M5 Ultra Mac Studio with 256GB memory, 8TB storage is $14,299
- NVIDIA RTX PRO 6000 with 96GB is street price $14,999. Sill need a $3K to $5K system to run it.
The NVIDIA card supports full CUDA and will be 2-4x faster at prefill, but for many people the Mac Studio will great for local AI development.
Also one big advantage of the studio is you can have it at your desk or in your office, most people don’t want the NVIDIA system’s noise and heat in their office.
Show more
Good Morning ! 80c 256 2TB
Would ya look at that ? Just look at it.
100% Throw-away, Wont merge this to main, it's just a concept.
MLX-Serve now can compile for Windows & Linux. (cli only)
It leverages CUDA/GGUF lib to do a few things, across the network.
1. Use Nvidia GPU for Prefill - Works well and clear win over Mac even over wifi.
2. Load big models across all your Mac/Linux/Windows nodes. - SLOW, not good.. this is why you need RDMA (duh)
3. Load balance multiple users across the ring network
4. Offload DiT processing from MiniMax H3 / LTX on Nvidia. (Text, VAE, Audio, Mux stays on Mac) - No code for this yet, just an idea.
Learnings: Maybe just port the prefill + cross platform parts, and simply leverage LLama.cpp and Comfy UI api's for others, to keep minimalistic code.
Thanks @djnmrjnvc for the idea on Prefill, it works well.
Show more
Qwen 3.8 - 27B MLX-Serve 4bit model, made the best version of the famous Pagoda test so far for me. Using @pidotdev as the coding agent, and MLX-Serve as the backend.
