VulcanBench
@VulcanBench
Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.
加入 March 2020
73 正在關注 2.8K 粉絲
Ruh roh, Muse Wonder could be the most expensive model to benchmark that we've ever come across.
The cost-per-task is looking incredibly high 😮 🍉
And this is also why it's soooooo important to not just show accuracy. It's the combination of accuracy and cost-per-task that matters if you want to pick a model for daily use.
顯示更多
Out of the gate, I am finding Muse Wonder to be a very expensive model to use.
At Low Effort, I hit my weekly usage limit on my Muse Code plan in less than 12 hours.
With VulcanBench, I couldn't get through an entire effort level run without hitting my limits.
Compare this to even something like Astra, which did every effort level, in under 12 hours, for less than $50.
That makes Muse Wonder more expensive, and a lot slower, than Astra.
Now I'm not sure what to do. I'll either needs to pay a small fortune to benchmark this model given how token hungry it is, or take a month or longer benchmarking it.
Put together some preliminary results comparing Muse Wonder to Opus 5.5 and GPT 6.1 Sol w/respect to token use.
While it's looking like an accurate model, I care a lot about cost-per-task, as we all should, and right now, this number is looking very high.
顯示更多
