VulcanBench
@VulcanBench
Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.
Joined March 2020
71 Following 2.7K Followers
Three generations of Sol, and I'll just come out and say it, very impressed with GPT-6.1 Sol, this is an exceptional model.
More details below:
Okay, this is definitely one of the most interesting benchmark comparisons I have ever done.
GPT 5.6, 6.0, and 6.1 Sol, across all effort levels on @VulcanBench Frontier v4.
And honestly, I don't think I even need to give any analysis or explanations because the results really do speak for themselves.
But one thing I will highlight is that GPT-5.6 Sol and GPT-6.0 Sol all struggled at Low effort, GPT-6.1 is a beast at Low effort.
I am now convinced, GPT-6.1 Sol is a very good model, and you can comfortable use it, at Low effort, and see strong, insanely cost-effective results.
I don't say this often but I can tell you, after seeing this data, I'd be crazy not to make this one of my daily drivers.
Full details from the benchmark now live on the site:
Live long and benchmark 🖖
Show more
