VulcanBench
@VulcanBench
Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.
Joined March 2020
71 Following 2.7K Followers
Early read on our GPT 6 Sol benchmark.
Okay, getting ready to share my GPT 6 Luna benchmark results, and wanted to also share what I have so far for GPT 6 Sol.
What I think is interesting is comparing GPT 5.6 Sol to GPT 6 Sol.
Not quite what I would have expected on the token/cost efficiency side, but I am very curious what OpenAI will announce at Dev Day today. Another reason why I wanted to get these results out on the sooner side!
Since I don't get early access like the cool kids, I have to start my benchmarks pretty late, and with the pace things are moving, this makes things pretty complex!
Early read below, full report to come once Max is finished.
Live long and benchmark 🖖
Show more
