註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Morgan
@morganlinton
cto @boldmetrics, evals @vulcanbench / not an expert, endlessly curious
加入 January 2009
1K 正在關注    47K 粉絲
Okay, results are in, my full benchmark on @VulcanBench Frontier v4 of GPT-6 Luna. While with many models I'm happy with Low or Medium effort, with Luna you do really need Extra High or Max effort to get meaningful performance. That being said, I think anything around 80 on my benchmark is more than good enough for easy and medium difficulty tasks. It's important to note that Luna should not be seen as a competitive model to Opus 5.5. On Frontier v4, Opus 5.5 Low Effort beats GPT-6 Luna at every effort level. But, as I've said many times, you don't need the worlds most intelligent frontier model to do all of your routine coding tasks, hard stuff yes, but common, you're not doing ultra-hard stuff all day, be honest. Of course, the real question is, how does GPT-6 Luna compare to GPT-5.6 Luna, and well, the reality is, it's not as accurate, but it is cheaper. I do think this means that there's more to dig in here. At the end of the day, making Luna cheaper, and slightly less accurate might actually be a very good thing for all of us. This is where my new Routine Eval Suite should hopefully help us decode this more. I think people jump to conclusions looking at accuracy reports on benchmarks meant to stump the frontier, when most engineers aren't doing frontier-stumping work on a daily basis. More to come, I definitely want to dig deeper with Luna, I think it's a pretty interesting model. You can do a deeper dive into the data on the VulcanBench site here:
顯示更多
0
15
51
5
轉發到社區