So this morning I sat down and analyzed the traces of my Beam benchmark. I wanted to know why it scored so poorly on coding tasks. I learned two core things that Beam did differently from models like Opus, Sol, and Astra, which is why it scored so differently from these models. Here's the difference when you look at the traces and see how Beam approached solving tasks in Frontier v4. 1. It made up a time limit and quit. - Beam’s instructions set no time or budget limit; none of those words appear in them. But, looking at the traces, it's reasoning says things like “I lack time” and “This is too deep. I’ll stop”, then it leaves the code unchanged. - At Low its median was 11 commands. 18 of 23 Low runs end by admitting it stopped short, with lines like “beyond this session’s budget” or “No changes made”. - Two runs blamed the sandbox: the old program just wasn’t marked executable, and a one-line chmod fixed it. At that point Opus 5.5 solved 17 of 23 and Astra solved 22 in about 4 minutes. 2. At higher effort it kept going, but its checks were too weak. - From High up it ran 100 to 500 commands, mostly sending inputs through the old program and comparing outputs. - Its own random tests reported no differences, but they didn’t cover the inputs the hidden tests use. Settlecore at Max ended with “All fuzzers pass” and passed 0 of 10 hidden behaviors. Its last test inputs were mostly random bytes, which only exercise the input-rejection path, not the fee rules. - The frontier models don’t stop until they have matched the behavior the task describes. If anyone else wants to go through the traces, as you all know by now, it's all available. Everything I do with VulcanBench is free, and open source. And yes, this means VulcanBench is the biggest money-losing idea I've ever had. But also the most satisfying. It's really nice to be a truly independent benchmark where I can just run experiments, and share the results with the world, and then anyone can examine or re-run anything they want to! So on that note, here's the traces if you want to do a deep dive yourself: Live long and benchmark 🖖