VulcanBench
@VulcanBench
Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.
๊ฐ์
March 2020
71 ํ๋ก์ ์ค 2.7K ํฌ
Great analysis from Michael! ๐
So I went task by task for the recommended medium level and max for comparison.
Sol and Astra clearly look alike. Similar trend with increased effort with more cost and more time. Sol 6.1 looks like a cheaper slower Astra.
Opus 5.5 results are wild for 2/3 of the tasks it is cost competitive. However it doesn't have any effort trend and some tasks are all over the place in cost.
Still, all are major improvements over predecessors.
Kudos to @morganlinton and @VulcanBench
๋ ๋ณด๊ธฐ
