Okay, the @VulcanBench results for Qwen3.8-Max are in, and it is not what I expected. First, for anyone new to VulcanBench, here's a quick TL;DR on the eval suite: 23 frontier-hard software engineering tasks taken from real merged OSS PRs, run in a Docker sandbox, 3 runs per task across all three of its effort levels. No puzzles, no random abstract stuff, all real things engineering teams would do with these models. It looks like Qwen3.8-Max has a major overthinking problem, it uses a LOT of tokens and is very slow, period, no other way to see it. My cost to run this benchmark was $126.25, to run the exact same eval suite with DeepSeek V4-Flash was only $13.60. This makes Qwen3.8-Max an insanely expensive model. The tasks Qwen genuinely can't solve fail at every effort level, extra reasoning didn't help. The regression is almost all in work it already handles: six tasks that low solves every single time account for 83% of the 26-point drop, three of them collapsing to zero. It's not losing the hard problems. It's losing the ones it already knows how to do. Since Qwen3.8-Max hit a lot of wall clock budget caps, I thought I'd share more about this. - VulcanBench caps both steps (50–200) and wall clock (5–60 min), each scaled by repo size. - This is aligned with how comparable harnesses bound agents, DeepSWE caps rollouts at 100 environment steps, sitting right inside my step range; Terminal-Bench enforces a per-task wall clock; SWE-bench Verified scaffolds typically allow 20–60 min per instance with 250–350 step limits. - Every model on my chart gets the identical budget, and Qwen is the slowest model I've tested at 20–25 min/task. Soooo... Alibaba positions Qwen3.8-Max as trailing only Claude Fable 5. But on the kind of real coding work engineering teams would actually throw at it, under a fixed budget, its best setting lands mid-pack and its default lands last, so common. If you want to optimize for accuracy, Grok 4.5 is the move. If you want accuracy per dollar, DeepSeek V4-Flash is hard to beat, heck it's 10× cheaper than Qwen and you get higher accuracy. Qwen just isn't in the game at this point, this is not a model I could see engineering teams using for daily coding work.

I've been using Grok Build since the first 24-hours it was made available. It is still my favorite agentic coding cli, and there are so many little features I just love. So I thought I'd put together a little thread on what makes Grok Build so special, and some features you should try if you haven't yet. Let's kick it off 🧵

Loading up the APIs for my next @VulcanBench run. Put a lot into this one, v3 eval suite with 23 new evals. Broader language coverage. And some really deep dives into making sure this doesn't have any of the issues that OpenAI raised yesterday with SWE-Bench Pro's four failure modes. Still, I'm new to benchmarking, so there's always a chance that I waste like $150 here. Hoping that's not the case, feels like I've tried to think of every edge case. This is the comparison I've been so excited to do: Grok 4.5 vs. GPT 5.6 Sol vs. Fable 5 All on Low and Medium effort, across real tasks that represent the work real engineering teams will be doing daily. No puzzles, no weird math problems, and looking not just at accuracy, but token use, cost, and time. In aggregate this will be a total of 138 runs, will wake up on Saturday to see the results.

Quick question @ArtificialAnlys, in this chart of cost per task, what effort level are you using with Fable 5? I've found that Fable Five Low and Medium effort can match the accuracy of most other frontier models on the same task for real engineering work. So I'm wondering if this chart gives an accurate cost per task if Fable 5 could have still performed well across these tasks at a lower effort level, but a higher effort level is used across tasks when it's not needed. Genuine curiousity!

FWIW, it looks like it's not just @mattshumer_ that ran into this issue 👀 Quick reminder, you can use hooks and setup a hook to stop GPT 5.6 from running rm -rf Docs are here:

Today is a good day to remind everyone that hooks exist, and you should probably be using them, esp. to prevent new models from accidentally deleting all of your files. Sadly this happened to @mattshumer_ with GPT 5.6 Sol, scary stuff! 😳

So I'm pivoting my game dev plan, and getting more excited in the process, here's the update: 1. I think I want to write 50%+ of the code for my game. Maybe I'm just old, but I've been coding for 30 years, and while I'm okay with AI writing a solid chunk of code for the game, I want to write most of it, and I want to understand all of it. 2. I'm going to use Unity rather than Godot. I've been building in Unity for 5+ years, and I really like C# and have written a lot of C# code for Unity games, so I think I'll stick with what I know. 3. I'm going to use a combo of models, I think this will be a fun way to see what model shines across different tasks. The three core models I'll be using are: Grok 4.5, Fable 5, GPT 5.6 Sol. 4. I will be using LLMs as my game dev tutor. We'll be do 30 minute sessions together every day, that's about all the bandwidth I'll have so it'll have to do. Put together a plan with Fable 5, it's been my favorite planning model so it will be my tutor to start. 5. I will be keeping a (light) devlog, and will share the link to this once it's up, might even try to record a video or two if I can. I absolutely love watching indie game dev logs on You Tube, I've been doing it for over a decade, few people know this about me. Oh and I have GPT 5.6 Sol create a logo, I think it nailed it. Always loved this name. Felt crazy spending $5,000 on a domain name before I even had an idea for the game, but now I'm really glad I did, I guess I just knew this day would come.

I'm going to hold off on testing GPT 5.6 Sol in VulcanBench as there seems to be a small bug that might be impacting token use. Since VulcanBench scores models based on accuracy and token efficiency, I want to make sure to give GPT 5.6 Sol a fair shot. That being said, so far I am really liking the model, it's a great model, and the Codex team is always super fast at responding to and fixing stuff like this, so I still think there's a chance I can benchmark it this weekend!

Okay, my new 23-task eval suite for @VulcanBench is done. A lot of little details I wanted to get right with this. It was important to me that I had more broad language coverage, and also that it takes into account the issues that OpenAI raised yesterday with SWE-Bench Pro's four failure modes. I'm new to benchmarking, but learning more every time I create a new set of evals. I'm calling this set v3, getting ready to run the first smoke tests. Then this weekend, phew, safe to say I have a lot of models to test! I will be committing these to Github so anyone can take a look and give me feedback on these as well. Live long and Benchmark 🖖