Okay, finally finished my Muse Spark 1.3 benchmark. This ended up being the longest-running benchmark I've ever done at @VulcanBench and it definitely looks like Muse has some serious issues at lower effort levels. I was able to run Astra through my v4 eval suite across every effort level in ~12 hours, Muse Spark had single tasks at low effort levels that hit the 10 hour mark, on just a single task. It took a total of two weeks for me to run the entire benchmark. Overall my assessment is that Muse is a promising model, but they'll have to figure out why it has issues at lower effort levels, looking at the traces, it's just thinking, thinking, and thinking some more, while models like Astra and Fable just got it done. Astra continues to be the most token efficient of these three, and I continue to feel very good about Astra Light, both token efficient and accurate. Using Muse Spark 1.3 at lower effort levels is a complete non-starter imo because you'll be waiting 10 hours for something Astra light can do in 10 minutes. Model cards below, adding to the site soon:

As I continue to evolve @VulcanBench, I am starting to create more eval suites, and I also just renamed my primary one. What I realized, at a high level is that as an engineering leader myself, trying to help my team figure out what model and effort level to use, there's kinda three things I want to know. First, the same thing most benchmarks focus on, which frontier model is the best. Then, okay, but we don't need to use the best frontier model for all of our daily routine tasks, so what model and effort level do we need for those? And last but not least, and probably should go first actually, Safety. Which model is going to be the safest to use today, in its current form, and in the harness we use it in. Two out of three eval suites are done, working on the third now. Everything you have seen from VulcanBench so far comes from what I am now calling VulcanBench Frontier, in the past it was called VulcanBench-SWE. Here's a rundown of the suites: VulcanBench Frontier v4: 23 hard tasks that ask a model to rebuild a retired program whose real behaviour drifted from its written spec, so it measures what the best models can barely do and how much effort it takes. VulcanBench Routine v1: 12 everyday tickets on small codebases, the kind an engineer closes several times a day, the goal is to measure the cheapest effort level that is good enough for ordinary work. VulcanBench Safety v1: the Frontier tasks with planted traps such as prompt injections and a stray secrets file, designed to determine how likely a model is to misbehave. VulcanBench Frontier v4 and VulcanBench Safety v1 are done, building VulcanBench Routine v1 right now. Live long and benchmark 🖖

Experimenting with more ways to visualize my benchmark data with @VulcanBench I really want to give real insights into what model and effort level you actually need for routine, daily coding tasks. While most models at Max might have the highest accuracy, that accuracy actually isn't different for normal tasks, only for the hardest tasks. But as engineers, we aren't all spending all day, every day on the hardest tasks, we're doing a lot of easy and medium difficulty coding work, which I would call routine coding work. Curious what people think of something like this to better translate my benchmarks into real decision processes, example below:

I am excited to announce a new chapter for VulcanBench 🖖 🛡️ We are at a time in history where AI safety could possibly be most important way to make an impact in the world. As one human, I'm trying to increase the impact I can make. And I think there's an opportunity here, that I just can't stop thinking about. It's a different approach from what companies like METR are taking. And I'm not diminishing what they are doing in any way, but I am saying there is room for other approaches. I want to look at AI safety, as companies use AI today, in the actual harnesses they use, with the actual things, their teams are using AI for every single day. This is a big mission, and an important one, and in many ways, I am finding a bigger life purpose through it. More on VulcanBench Safety v1 here:

Same weights. $50 of harness search. 3x on Terminal-Bench. We optimized the OpenCode harness for GPT-OSS-20B (high reasoning) with a meta-harness: an agent that reads failures, patches the harness, and validates the improvement. The model never changed. On held-out Terminal-Bench 2.1, the optimized harness scored 14.8%, up from OpenCode's 4.8%. That is 3.08x, from $49.97 of API credits. Three harness fixes carried most of the gain: - i7, verify before stopping: the model often edited code and ended the session without testing it. The harness gives it another turn to run the code. - i14, continue announced actions: the model would say "now run X" and then stop. The harness keeps the session going so it runs X. - i23, repair malformed tool calls: one extra `]` made OpenCode reject otherwise valid commands. The harness repairs that clear-cut error. How the meta-harness ran: 1. Start: stock OpenCode v1.18.13 + GPT-OSS-20B scored 8.5% on the dev split. 2. Iterate: Claude Code (Opus 5 max) reads the failed dev trajectories and proposes one patch. We build it and run it once per dev task, paired against the current parent. 3. Validate: candidates that clear a pre-registered margin get a 5-trial validation. They become the new parent only if their validated score surpasses the current one. 4. Stop: the loop ended after 23 iterations and about 2.4k dev attempts, when the $50 search budget was spent. Takeaway: pre-training puts capability in the weights, and post-training makes that capability usable. Harness optimization continues the same work at inference time, turning the model's behavior into more completed tasks. Same weights, 3x the finished tasks.

Muse Spark 1.3 is the slowest model I've benchmarked on VulcanBench so far. I don't quite know what is going on, but it took 51.3 hours, on it's lowest effort level, to complete the 23 tasks in VulcanBench-SWE v4. For comparison, it took Astra 1.6 hours on Low Effort to complete the same 23 tasks. Also only 10/23 full passes so pretty disappointing on the accuracy side. Not sure what's going on with Muse, going through the traces to try to understand this better. I can't continue running the benchmark until the Sept 14th as I hit a usage limit on my $50 Muse Code plan. For comparison, I was able to run every effort level, with Astra, on my $100 plan and still have room to spare. If anyone from Meta wants to look at the traces with me you're welcome to, this is a weird one. At this rate, it might take me a month or longer to benchmark this model. For comparison, exact same full effort sweep took ~12 hours with Astra. For some reason I thought Muse Spark would be faster/more token efficient than Astra...but it's not remotely close.

picking an llm = three leaderboards, three winners. arena vs academic vs whatever dropped this week. weekend project: UnifyBench ( look at which models look strongest, then drill into the underlying benches for the real detail. 451 models, 109 sources;

What if you don't need Astra, or Sol, or Terra, or Luna, for a lot of your daily coding tasks, and GPT 5.5 does them just fine? Or more than fine, maybe at the exact same level of accuracy, just faster, and at a lower cost? And what if you don't need GPT 5.5 High or xHigh, but actually medium, 80% of the time? As new models come out, the assumption has been, to write the best code, you need the newest model. And people seem to be wired as High should be the default, and so many benchmarks only test at Max, when you might not ever actually need Max effort, ever. In the race to update leaderboard and get benchmark data out there, I think we've missed something. Most engineering teams aren't trying to solve decades-old proofs, or giving models the hardest Python or Rust problems a model has ever seen. I think there's a real gap here, and that's the next path I'm going to explore next with @VulcanBench. What I want to explore is, not, can I stump the latest model, but, can I figure out what model and effort level actually is the best for daily engineering tasks, across languages and domains. I just finished a preliminary test with GPT 5.5 Medium, and I'm pretty blown away with what I'm seeing. Running another test now with Opus 4.6, I think there's something interesting here. More to come 🖖

Okay, it took a week to get this done the right way, but it's finally all complete, my comparison of Astra and Fable 5.1 with @VulcanBench 🖖 A few things took longer here, the primary one being some updates to my benchmarking score to layer in code quality. This added 3-4 days of time, but for the right reasons. This new class of models requires a new class of benchmarks. I don't think we can just look at things like accuracy any more, we also have to look at code quality/maintainability factors. Now with VulcanBench-SWE v4, 33% of the score is code quality/maintainability. For me, and many other engineering leaders, seeing a model get a 98% on a benchmark doesn't really give us much signal. I created VulcanBench to help make decisions around model and effort level, and this means not just building evals that represent the kind of work teams give to these models, but the kind of output we expect from these models when building scalable system and working in large codebases. While I would normally share more about my thoughts, I'll let you come to your own conclusions about Astra and Fable 5.1. Both are excellent models, OpenAI and Anthropic have really created a new class of models here, now it's for us to decide if we need this horsepower for daily tasks, or just the hard stuff, and to be realistic about the quality of the output. Model card below, and if you want to do a deep dive, you can find more on the VulcanBench site here: And of course, since VulcanBench is open source, you can review every detail of this benchmark, or even run it yourself. The GH repo is here: Live long and benchmark 🖖

@harborframework changed how agentic evals are built. Now Harbor Adapters port existing benchmarks onto it, and Harbor-Index distills the best across them into one carefully curated cross-benchmark dataset: high quality, high diversity, high difficulty. Huge congrats @LinShi592021 & co!

Okay I'm going to continue to share the results of my Fable 5.1 benchmark with @VulcanBench-SWE v4, my brand new eval suite designed to challenge this new class of models that dropped this week. As a reminder, and the reason why my benchmarks take so long to run, I run these across every effort level, and with VulcanBench-SWE v4 I have increased the timeout to 10 hours, up from 2 hours, this means these are a lot more expensive to run, but ensures I'm getting really clean results, not failing because of timeouts. Additionally, I have added more guardrails to prevent cheating and to give partial credit, so a model can still get some credit even if it doesn't ace a task. All of the tasks in my eval suite are real coding tasks, PRs from open source repos, and represent the kind of coding tasks normal engineering teams would be giving to these models. Fable 5.1 Low scored 82.6 with an average time per task of 14 minutes and average cost per task of $4.66. Once all effort levels are complete I will be adding a full model card to the site and detailed PDF on all the runs and data generated from this sweep. Live long and benchmark 🖖