註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

VulcanBench
@VulcanBench
Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.
加入 March 2020
63 正在關注    2.3K 粉絲
Seeing a high volume of posts on X about Astra being nerfed. Putting together a plan to test this with VulcanBench to move from anecdotal evidence to a more data-driven approach. Since I benchmarked it launch week, I do have data to compare how it performed at launch vs. now. But VulcanBench only looks at coding, not things like 3d rendering, game design, etc. and a lot of the nerf reports seemed to be focused on this so I might be somewhat limited. Still, worth seeing if we can notice any kind of difference. Let's let the data do the talking and answer the question: is Astra really nerfed?
顯示更多
🚨 GPT-6 Astra (Launch) Vs GPT-6 Astra (Today) Same exact prompt But today’s result looks noticeably worse It genuinely feels like OpenAI nerfed Astra