VulcanBench(@VulcanBench):New on VulcanBench, our Analysis section debuts with a look into the traces for Muse Wonder. At Low effort, Muse Wonder seems to have an overthinking problem. To better understand what is happening here, we compare Muse Wonder with GPT-6.1 Sol on the same tasks and compare how each model thinks differently. Overall Muse Wonder takes more than 4x the time to complete a task vs. GPT-6.1 Sol. Full trace analysis below. Live long and benchmark 🖖

VulcanBench

@VulcanBench

Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.

参加 March 2020

73 フォロー中 2.8K ファン

VulcanBench@VulcanBench

2026.10.14 08:04

New on VulcanBench, our Analysis section debuts with a look into the traces for Muse Wonder. At Low effort, Muse Wonder seems to have an overthinking problem. To better understand what is happening here, we compare Muse Wonder with GPT-6.1 Sol on the same tasks and compare how each model thinks differently. Overall Muse Wonder takes more than 4x the time to complete a task vs. GPT-6.1 Sol. Full trace analysis below. Live long and benchmark 🖖