Okay, results are in, my full benchmark on
@VulcanBench Frontier v4 of GPT-6 Luna.
While with many models I'm happy with Low or Medium effort, with Luna you do really need Extra High or Max effort to get meaningful performance.
That being said, I think anything around 80 on my benchmark is more than good enough for easy and medium difficulty tasks.
It's important to note that Luna should not be seen as a competitive model to Opus 5.5. On Frontier v4, Opus 5.5 Low Effort beats GPT-6 Luna at every effort level.
But, as I've said many times, you don't need the worlds most intelligent frontier model to do all of your routine coding tasks, hard stuff yes, but common, you're not doing ultra-hard stuff all day, be honest.
Of course, the real question is, how does GPT-6 Luna compare to GPT-5.6 Luna, and well, the reality is, it's not as accurate, but it is cheaper.
I do think this means that there's more to dig in here. At the end of the day, making Luna cheaper, and slightly less accurate might actually be a very good thing for all of us.
This is where my new Routine Eval Suite should hopefully help us decode this more. I think people jump to conclusions looking at accuracy reports on benchmarks meant to stump the frontier, when most engineers aren't doing frontier-stumping work on a daily basis.
More to come, I definitely want to dig deeper with Luna, I think it's a pretty interesting model.
You can do a deeper dive into the data on the VulcanBench site here: