Learning about the pros and cons of local ai in realtime here benchmarking Glimmer. This will be the first benchmark I run where the cost is $0...but also my first benchmark that will take 1-3 days to complete 👀

Okay, well since I can't get access to Muse Spark through the Meta API yet, I'm going to benchmark Muse Glimmer with @VulcanBench 🖖 This will be the first model I test that has a cost of $0, which I like a lot! If you don't know about Muse Glimmer, it was just announced by @finkd this week. Major step forward for American models, and something that I should be able to run locally on my four year old M1 Mac Studio, which is pretty darn cool.

If you are using one model, at high effort, to plan, and code, I can tell you that you’re likely spending a lot more than you need to, and moving a lot slower. Plan with one model, code with two or three different models, some at low effort, others at medium, and use high effort way less, you don’t need it as much as you think. And pretty much forget extra and max exist. Seriously.

I have now had more than a few companies reach out to me about doing stack-specific benchmarks and model reports with @VulcanBench. This wasn't something I thought of, but then it kinda clicked. I'm also hearing now more and more about challenges eng leaders are having taking all the benchmarks that are out there, and figuring out how to apply them to their teams, and what kind of model routing might be best for them based on their stack and codebase size. This is also where effort levels really matter. I've heard from a lot of people that, "my team just uses one model, high effort, for everything." And yeah, that's not optimal. It's both expensive, and slow, and you'll end up with your eng team waiting around for hours, when they could get a task done faster, and cheaper, if they had better model routing. This is different from all the model routers companies are coming out with, because those are general, with VulcanBench, I can get really specific using actual PRs from real work that eng teams are doing to customize model routing just for them. And, since I now have a growing list of companies that want this, I guess it just makes sense to add it to the VulcanBench site and start offering it as a service, and we'll just see where it goes! No idea how to price it, just putting it there as an email link and will take it convo by convo. You can read more about it here: