Okay so I managed to squeeze in getting model cards out today for my Muse Wonder Full benchmark with
@VulcanBench - enjoy!
This, plus all the data from every run will be on the VulcanBench site soon(ish), and all of the traces will be on Github as well.
Everything I do with VulcanBench is free and open source. As a truly independent benchmark (it's just me!) I share it all, every single trace from every run, plus the source code for VulcanBench so you can even re-rerun specific tests, or the entire benchmark if you want to.
What stands out looking at this is that Astra is fast, but you pay for it. GPT-6.1 Sol is fast and cheap. Opus wins in accuracy, and Muse Wonder is accurate, but expensive and slow.
Right now, with the data I have, I don't think I would use Muse Wonder for coding, and I'm not sure that's the market they're going after, yet.
That being said, looking at the accuracy numbers, it's clear the Muse team has built a very intelligent model. If they can figure out how to wrangle the time/cost per task, they could very easily become a model usable for daily coding tasks.
Live long and benchmark 🖖