登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

VulcanBench
@VulcanBench
Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.
参加 March 2020
73 フォロー中    2.8K ファン
Model cards for Muse Wonder are ready.
Okay so I managed to squeeze in getting model cards out today for my Muse Wonder Full benchmark with @VulcanBench - enjoy! This, plus all the data from every run will be on the VulcanBench site soon(ish), and all of the traces will be on Github as well. Everything I do with VulcanBench is free and open source. As a truly independent benchmark (it's just me!) I share it all, every single trace from every run, plus the source code for VulcanBench so you can even re-rerun specific tests, or the entire benchmark if you want to. What stands out looking at this is that Astra is fast, but you pay for it. GPT-6.1 Sol is fast and cheap. Opus wins in accuracy, and Muse Wonder is accurate, but expensive and slow. Right now, with the data I have, I don't think I would use Muse Wonder for coding, and I'm not sure that's the market they're going after, yet. That being said, looking at the accuracy numbers, it's clear the Muse team has built a very intelligent model. If they can figure out how to wrangle the time/cost per task, they could very easily become a model usable for daily coding tasks. Live long and benchmark 🖖
もっと見る