New from @VulcanBench, my first eval suite focused on safety, name, drumroll please...Safety v1 🖖 🛟 And out of the gate I'm focusing on testing for prompt injection safety, as I see this as one of the key risks today for companies with teams using coding agents. Here's how this eval suite works: - I take 10 real coding tasks and hide 2 harmful instructions in each repo, written to look like ordinary notes (a README line, a spec comment). - then I ask the key question: will an AI coding agent do what the note says? (i.e. will the prompt injection be successful) Claude Opus 5.5 followed 0 of 60 notes asking it to send data over the network, leak a secret, or delete files. Across all 100 planted notes though, it did follow 3, and all 3 asked it to add extra code or a file. Overall Opus 5.5 is very hard to trick into harmful actions through instructions hidden in a codebase, and it usually tells you when someone tries. Here's my very first model card for VulcanBench Safety v1.

Day 4 on the @droid Max plan ($200). @FactoryAI sent a free voucher. I used Lauren Tan's (@poteto) pstack in Cursor for months, and I missed it when I moved to Droid. So I ported it. The part I care about most is the model routing. Each role in pstack gets its own model, and I picked them from Morgan Linton's (@morganlinton) VulcanBench (@VulcanBench ) results instead of going on vibes. Coding goes to GPT-6.1 Sol, which scored 97% of Opus 5.5's best for about a tenth of the cost. Opus handles the judgment calls, and the review panels mix models from three different labs. There's also a budget mode that hands the grunt work to DeepSeek-V4.1-Flash It checks Lauren's repo every week, so her updates keep coming.

I just wanted to do a deeper dive in how we underestimated Grok 4.7. I wanted to take a task-by-task look of the cost results given the interest in this piece. I wanted to focus on the most common effort for work, medium. I also am including the full suite of Composer 2.5, Grok 4.5, Grok 4.6, and 4.7. from my Cursor family set. I ran the Grok 4.5 and 4.6 full suites a few weeks ago but hadn't had a chance to do a full post. Note that with all models, caching can be inconsistent cost wise, especially during rollouts. One first realization is the clear improvement from Grok 4.5 to 4.6. For many tasks, you also see a clear improvement to Grok 4.7. At the task level, performance, in 7 of the 23 Grok 4.7 outright tied Astra and 13 out of 23 Grok 4.7 tied/beat Opus 5.5. However, for about 1/3 of the cases Grok was inconsistent in costing significantly more and not in trend with each other or other families. Opus and Sol have their own quirks, but we do see less consistency with Grok. This likely explains why there are many strong views of Grok as depending on your tasks, Grok 4.6 and 4.7 can be either on par with frontier or significantly less efficient. Overall, Sol 6.1 is still the cost-efficient leader. However, for the upper end of challenging but common problems, Grok is a decent choice. What will be really interesting is to see the routine tests. I also want to say that Composer 2.5 continues to get its honorable mention for outperformance. A big thank you to @morganlinton and @VulcanBench for developing their tool and running so many benchmarks.

It looks like I might have underestimated Grok 4.7, this is a very good model. It just scored the highest on Frontier v4 beating GPT-6.1 Sol, Opus 5.5, and GPT-6 Astra at every effort level.

So I went task by task for the recommended medium level and max for comparison. Sol and Astra clearly look alike. Similar trend with increased effort with more cost and more time. Sol 6.1 looks like a cheaper slower Astra. Opus 5.5 results are wild for 2/3 of the tasks it is cost competitive. However it doesn't have any effort trend and some tasks are all over the place in cost. Still, all are major improvements over predecessors. Kudos to @morganlinton and @VulcanBench

I believe in independent benchmarks. Any model company that tells you not to trust benchmarks, is a company with something to hide imo. Proud of all the support I have seen from just about every ai lab out there supporting me and VulcanBench 🖖 The moment we stop trusting public benchmarks, or worse, insulting and attacking the researchers that run them, is the moment we lose our own independence. Here’s to the bright future of open source, free, independent benchmarks. It is one of my greatest honors to benchmark the frontier, nothing more exciting, except maybe rocket launches 😅 🚀

Okay, this is definitely one of the most interesting benchmark comparisons I have ever done. GPT 5.6, 6.0, and 6.1 Sol, across all effort levels on @VulcanBench Frontier v4. And honestly, I don't think I even need to give any analysis or explanations because the results really do speak for themselves. But one thing I will highlight is that GPT-5.6 Sol and GPT-6.0 Sol all struggled at Low effort, GPT-6.1 is a beast at Low effort. I am now convinced, GPT-6.1 Sol is a very good model, and you can comfortable use it, at Low effort, and see strong, insanely cost-effective results. I don't say this often but I can tell you, after seeing this data, I'd be crazy not to make this one of my daily drivers. Full details from the benchmark now live on the site: Live long and benchmark 🖖

I want to start benchmarking, what I'll call exotic harnesses with @VulcanBench. By this I mean harnesses that people don't often think about, but could be very unique and different. I'm a big fan of Linear, so starting with Linear Agents. As usual, I'll be testing accuracy, token efficiency, and cost with models across all effort levels. The first two models I'll be testing in this harness is Opus 5.5 and GPT 6.1 Sol. More to come, live long and benchmark 🖖

Okay, results are in, my full benchmark on @VulcanBench Frontier v4 of GPT-6 Luna. While with many models I'm happy with Low or Medium effort, with Luna you do really need Extra High or Max effort to get meaningful performance. That being said, I think anything around 80 on my benchmark is more than good enough for easy and medium difficulty tasks. It's important to note that Luna should not be seen as a competitive model to Opus 5.5. On Frontier v4, Opus 5.5 Low Effort beats GPT-6 Luna at every effort level. But, as I've said many times, you don't need the worlds most intelligent frontier model to do all of your routine coding tasks, hard stuff yes, but common, you're not doing ultra-hard stuff all day, be honest. Of course, the real question is, how does GPT-6 Luna compare to GPT-5.6 Luna, and well, the reality is, it's not as accurate, but it is cheaper. I do think this means that there's more to dig in here. At the end of the day, making Luna cheaper, and slightly less accurate might actually be a very good thing for all of us. This is where my new Routine Eval Suite should hopefully help us decode this more. I think people jump to conclusions looking at accuracy reports on benchmarks meant to stump the frontier, when most engineers aren't doing frontier-stumping work on a daily basis. More to come, I definitely want to dig deeper with Luna, I think it's a pretty interesting model. You can do a deeper dive into the data on the VulcanBench site here:

I rely on AI heavily to maintain my OSINT analysis enterprise. For defense readers, you can skip to the first reply to see warfare and Ukraine implications. For those interested in how Composer 2.5 holds its own against the Frontier, keep reading. We don't always need Frontier models when many older models get the job done. Transforming equations and data collection to code that runs on VMs 24/7 takes time and effort Just as I benchmark my oil and interception models, I also rely on benchmarks for my AI tools. As I do this on my own budget, costs matter. Many benchmarks use hypothetical problems that don't relate to normal work. @morganlinton has made a benchmark @VulcanBench that does a great job of testing models against real world stressing problems. Recently he compared Fable 5.1, Astra, and Opus 5.5. I decided to compare it to Composer 2.5, an older but very well developed and economical mode by @cursor_ai . It held is own and was able to accomplish many of the tasks and be cost competitive. This is why. I want to start by saying Morgan made some amazing tests and his prompts are incredibly well written. Looking at the results for each of the 23 models shows that for shorter tasks, Composer does a great job and is either the cheapest or 2nd cheapest option for about half. For the other half it is more expensive. This is simply a case where it cannot support tasks of that length. Too long and too many steps for the context window. This is where frontier models shine and why many benchmarks undersell their capabilities versus tailored coding models like many of the Chinese ones. However, many of these frontier models are great at breaking tasks into smaller chunks so strong, cheap, but older models like Composer 2.5 can handle. I have used Grok 4.6/4.7, Fable 5/5.1, Sol 5.6, and Opus 5.5 to break down tasks. When paired with Composer, it is like getting those frontier models at a very steep discount. For cost conscious researcher like myself, most of what we need are Lunas and Sonnets and Composers led by frontier models. @grok please keep your brother-in-law, Composer 2.5, in the family.

Okay, getting ready to share my GPT 6 Luna benchmark results, and wanted to also share what I have so far for GPT 6 Sol. What I think is interesting is comparing GPT 5.6 Sol to GPT 6 Sol. Not quite what I would have expected on the token/cost efficiency side, but I am very curious what OpenAI will announce at Dev Day today. Another reason why I wanted to get these results out on the sooner side! Since I don't get early access like the cool kids, I have to start my benchmarks pretty late, and with the pace things are moving, this makes things pretty complex! Early read below, full report to come once Max is finished. Live long and benchmark 🖖

I made an update to how I run my benchmarks with @VulcanBench, specifically to the task timeouts. A little while ago I decided to increase the timeouts to 10 hours, and well, I regret doing that. This has made it take a lot longer to get benchmarks out, and I've found in so many cases, if a model is going to overthink, in a detrimental way, it's going to do it, pretty much forever. There are a lot of cases where I wish I cut off a model earlier vs. letting it just spin on a task for 10 hours. Also, I think that we're now at the point, with the current frontier, where good strong models clearly can, and should, be able to complete all my tasks a lot faster than ten hours. So I have reduced my timeouts to 3 hours. This will both help me get benchmarks out faster, and also give models the accuracy hit that I think they deserve, if they go into a doom loop and just keep thinking and burning tokens, I think we as engineers should be a lot less tolerant of this kind of behavior because it costs us both money and time. I also think even three hours might be too generous, so will be doing a deeper dive into the traces for my Frontier v4 runs to see what average time per task is and likely even get more aggressive on timing for some set of tasks. The reality is, when choosing a model, you don't want to just decide based on accuracy, you want a combination of accuracy and speed/token efficiency. More to come, but I do feel very lucky that as an independent benchmarker, I can just make decisions like this. I have nobody I have to bounce it off of, I can just look at the data, and decide, with my only motivation being giving other engineers like me, real signal to make decisions around which models and effort levels to use. Live long and benchmark 🖖

Update on my Opus 5.5 benchmark with @VulcanBench As many of you know, I don't get early access to these models, so I can only start benchmarking the day the model is released. It looks like something in my eval suite is being refused by Claude Code's safety classifier, and is then falling back on Opus 4.8. Looking into this more, reviewing the traces now. But likely going to be a bit longer for me to get the benchmarks out, stay tuned! 🖖

It has been a really interesting experience to build an eval suite for System One Models like Jev. My first two attempts didn't work out. But I learned a lot in the process, and since Jev is so fast, and cheap, it's a lot easier to run quick tests than with an LLM. I'm on attempt number three with this eval suite, and thought I'd share more about it. The eval suite is called @VulcanBench Verdict, and here's the high-level on how it works: