Jim McPherson
@JamesMcPherson
Engineer specializing in RF, offensive cyber, and fabrication. ex USG. Tesla owner. FSD advocate. Believer. There are FOUR lights. W3RTR
3.1K Following 5.6K Followers
Currently the M5 Ultra is very much a competitor but don't count the Sparks out. They hold their own if serving more than one user.
Tonights testing got some very nice single user speeds using Tensorflow, but as expected prefill is important.
I ran 3 tests on long context windows.
1. M5 ultra+ Tensorflow
2. M5 Ultra plus oMLX.
3. 4x DGX Spark vLLM.
All on GLM 5.3 Flash.
The Sparks blow the M5 out of the water on cold prefill.
When in a coding session and the context window grows over time large context windows are fine on the M5, even with oMLX.
I'm still tracking down a serious bug with Tensorflow slowing down considerably after a long context window fills the ram. So these results may change after that gets squashed.
More to come.
Show more
First "Big" finding on the M5 Ultra.
TLDR: GLM 5.3 Code is running at 125 tok/s and prose at 87!
Earlier in the day, I optimized oMLX for GLM5.3 and peaked at a very respectable 57 tok/s prose which is competitive with my 4x Spark cluster.
After running Tensorflow and some tweaking for M5, (comments added to the PR for GLM5.3 so you can reproduce) I'm up to 87 tok/s prose and 125+ for code.
Now, this is not really fair as I haven't ported Tensorflow over to the 4x Spark stack yet, so I don't know what gains I'll see there. That's coming next.
Being able to load / unload / test / try so darn quick on the M5 Ultra turned what would have been a full days work into just a few hours thanks to the incredible speed of the M5 Ultra's RAM.
More to come but right now, the $10k for the Mac Studio is money well spent.
Show more
Anyone who uses FSD cringes every time they have to use Uber.
We paid $150 to Uber to get to the airport. Too many bags.
And $350 to get that Hummer ride into New York.
For two hours of driving combined.
And people wonder why Elon is building Robotaxi.
A self-driving Cybertruck would be cheaper and better experience.
The problem in New York is the unions.
They will hold out until the payday becomes big enough.
Or everyone else puts them in jail. How many tragedies will it take to change hearts and minds?
40,000 a year in USA alone.
Who will get the airport rides?
Hustler needed in line. :-)
Show more
First "Big" finding on the M5 Ultra.
TLDR: GLM 5.3 Code is running at 125 tok/s and prose at 87!
Earlier in the day, I optimized oMLX for GLM5.3 and peaked at a very respectable 57 tok/s prose which is competitive with my 4x Spark cluster.
After running Tensorflow and some tweaking for M5, (comments added to the PR for GLM5.3 so you can reproduce) I'm up to 87 tok/s prose and 125+ for code.
Now, this is not really fair as I haven't ported Tensorflow over to the 4x Spark stack yet, so I don't know what gains I'll see there. That's coming next.
Being able to load / unload / test / try so darn quick on the M5 Ultra turned what would have been a full days work into just a few hours thanks to the incredible speed of the M5 Ultra's RAM.
More to come but right now, the $10k for the Mac Studio is money well spent.
Show more
And people question the value of AI.
Thanks to @loktar00 for the tip that Microcenter has a screamer deal on these 4TB SSD’s.
I saved some money on my Mac Studio by going with the 2TB version. I couldn’t justify $3000 more just to go from 2TB to 8TB.
In an external TB5 enclosure they won’t be quite as fast as internal, but I can use the 2TB internal drive as “cache” while not relying on the NAS to store all the models and quants I will be flipping between.
Show more
It’s not the perfect day for many things, but it is the perfect day for the Cybertruck reflective rain jacket.
Cybertruck is the perfect vehicle to see the country from.
Nothing has given me more joy and adventure in the last year than owning @cybertruck
FSD + Starlink + offroad capability + incredible comfort in a tough stainless cocoon.
No other vehicle makes it so easy, no matter if you’re just commuting, road tripping across states or offroading in the beautiful backcountry.
Cybertruck makes it so you look forward to long journeys on the road instead of dreading them.
Hoping to tackle the rest of the country over time :)
Happy 1 year anniversary, chihiro <3
Show more
Bezos was a proven highly successful entrepreneur with more money who started earlier and hired the best talent he could find and let them do their thing.
Granted he didn't go all in the way Musk did with early investment, but on the whole he invested a lot more and over 2 decades later remains waay behind.
Blue Origin, by the way, would be a strong success story if not for comparisons to SpaceX.
Show more
They said it a lot.
My mind is blown 🤯! Loading AI weights on the M5 Ultra is FAST!
It's over 20 times faster than my Spark cluster.
When trying new recipes, the hard part is making the change then needing 12 minutes to reload.
Or even in every day usage, you pick the best overall model and are kind of "stuck" with it because it takes so long to load. If you want to do something different like image generation, you have to spend 12 minutes waiting to trade out the models.
The M5 Ultra loads 177GB of GLM5.3 in 25 seconds. WOW.
Show more
Super grateful for all the new follows recently. The AI community is on a thrill ride like never before and I'm happy to be holding on to handles as hard as I can.
Apologies if I haven't followed back yet, I'm trying to work through as many of you as I can!
Show more
I think there is a chance this plays out.
I decided not to sell my M5 Ultra because I sold my 3090 when I upgraded and that was dumb.
I have 4 Sparks that I averaged $3700/each for and they are now $8k.
Everything is trending up ad I have no idea if I'll ever be able to buy these again.
Show more
I think Sparks could hit $20k CAD / $14k USD each in the near term.
RTX 3090s? $3500 CAD / $2500 USD each.
5090? Closer to $15k CAD / $10k UsD each.
It sounds crazy.
But this is where we are at.
If your immediate reaction is not to smile ear to ear after watching this, I’m afraid we wouldn’t get along.
I'm actually working on doing this...
Hot news:
If you're a qualified purchaser, you can actually buy yourself a Bullfrog, RIGHT NOW, from
Even the @FCC is on X.
@Xmoney just made it official for my new General Amateur Radio license thanks to the Laurel Amateur Radio Club at @gnuradio conference .
Watch out for me on the HF bands soon!
Show more
This is why I'm not looking at how much my portfolio is down and I'm continuing to buy. Both SPCX and TSLA are gearing up for their next phase. It is a slow spot in earning, but it is NOT a slow spot in effort. They have a vision and are executing. That's what I'm investing in.
Show more
Huge storm just came through. Power is out for the whole neighborhood.
Except for me. Thanks Cybertruck.
Photo from moments before the bottom fell out.
