Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers. Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale. New to the Intelligence Index: ➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change ➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment Upgraded: ➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions ➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised ➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1 Key results: ➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50 ➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36) ➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04

So this morning I sat down and analyzed the traces of my Beam benchmark. I wanted to know why it scored so poorly on coding tasks. I learned two core things that Beam did differently from models like Opus, Sol, and Astra, which is why it scored so differently from these models. Here's the difference when you look at the traces and see how Beam approached solving tasks in Frontier v4. 1. It made up a time limit and quit. - Beam’s instructions set no time or budget limit; none of those words appear in them. But, looking at the traces, it's reasoning says things like “I lack time” and “This is too deep. I’ll stop”, then it leaves the code unchanged. - At Low its median was 11 commands. 18 of 23 Low runs end by admitting it stopped short, with lines like “beyond this session’s budget” or “No changes made”. - Two runs blamed the sandbox: the old program just wasn’t marked executable, and a one-line chmod fixed it. At that point Opus 5.5 solved 17 of 23 and Astra solved 22 in about 4 minutes. 2. At higher effort it kept going, but its checks were too weak. - From High up it ran 100 to 500 commands, mostly sending inputs through the old program and comparing outputs. - Its own random tests reported no differences, but they didn’t cover the inputs the hidden tests use. Settlecore at Max ended with “All fuzzers pass” and passed 0 of 10 hidden behaviors. Its last test inputs were mostly random bytes, which only exercise the input-rejection path, not the fee rules. - The frontier models don’t stop until they have matched the behavior the task describes. If anyone else wants to go through the traces, as you all know by now, it's all available. Everything I do with VulcanBench is free, and open source. And yes, this means VulcanBench is the biggest money-losing idea I've ever had. But also the most satisfying. It's really nice to be a truly independent benchmark where I can just run experiments, and share the results with the world, and then anyone can examine or re-run anything they want to! So on that note, here's the traces if you want to do a deep dive yourself: Live long and benchmark 🖖

Okay, my Beam benchmark is done, and well, the results aren't very good...but it's also important to remember that @VulcanBench is focused entirely on agentic coding tasks. Beam might be good at other things. I am excited for where open models are going, and am betting that companies like Reflection have a bright future. But there is one very interesting signal from this benchmark. Once you put Beam on a chart with models like Opus, Sol, and Astra, you see just how close the current frontier really is. But the signal here is clear, this is not a model I would use for coding. Full benchmark details will be added to the VulcanBench site soon. Live long and benchmark 🖖

Okay, just finished my DeepSeek v4.1 Flash benchmark run on @VulcanBench 🖖 And when I woke up to see the results, I kinda did a double take. You really have to look at the cost-per-task comparison in a bar chart to get it. This is an insanely cost effective model, I am impressed. For those who missed me saying this in the past. On my Frontier v4 eval suite, anything in the 80 - 90 range is more than good enough for routine daily work. The only weird thing about DeepSeek v4.1 Flash is that increasing effort level doesn't seem to really matter. Will be digging into the traces to understand this better as this is the only real head scratcher for me besides that fact that it's practically free to run this model. I mean seriously, look at cost-per-task, the green bar is GPT-6.1 Sol, the orange is Claude Opus 5.5 and the blue is DeepSeek v4.1 Flash. Everything is now live on the site so for anyone who wants to do a deeper dive, here ya' go:

Announcing Artificial Analysis Coding Agent Index v2.0: FrontierCode v1.1 from Cognition replaces DeepSWE, SWE-Atlas-QnA grading is revised, and Claude Opus 5.5 in Claude Code leads at 61.7 Today we are launching v2.0 of the Coding Agent Index. The Index evaluates models together with their agent harnesses, now across 290 tasks covering software engineering, terminal use, and repository Q&A, with the three benchmarks weighted equally. Changelog (Index v1.5 → Index v2.0): → FrontierCode v1.1 replaces DeepSWE v1.1. FrontierCode is a benchmark from @cognition with a private test set: 100 software engineering tasks on real open-source repositories, graded on whether a maintainer would merge the change. An attempt that fails a blocking check scores zero, and one that passes them all scores between 70% and 100% depending on the non-blocking checks it meets. We also scan each trajectory with a modified version of Cognition's rule-based scanner, and any attempt flagged for accessing solution material scores zero. → Revised SWE-Atlas-QnA grading, now SWE-Atlas-QnA-AA. Scale AI's 124 repository Q&A tasks are unchanged. We reworded 127 of the 1,306 rubric criteria for more consistent verdicts and moved to GPT-6 Luna (xhigh) as the judge, with five votes per criterion. The dataset and judge configuration are published on Hugging Face. Key results: ➤ Claude Opus 5.5 leads the Index: Claude Code with Claude Opus 5.5 (max) scores 61.7, followed by Claude Code with Claude Sonnet 5.5 (max) at 59.3, Muse Code with Muse Wonder Full (max) at 58.9 and Claude Code with Claude Fable 5.1 (max with fallback) at 57.6. ➤ FrontierCode v1.1 is where Opus 5.5 pulls ahead: Opus 5.5 scores 56.0% on FrontierCode, ahead of Codex with DeepSeek V4.1 Flash (max) at 52.6%, Codex with GPT-6.1 Sol (max) at 49.4% and Codex with GPT-6 Astra (max) at 49.3%. Below the top, FrontierCode scores are tightly packed: 15 of the 24 configurations on the Index score between 43% and 49%. ➤ SWE-Atlas-QnA-AA grading is stricter and more consistent: across the 21 configurations evaluated on both Index versions, the mean score falls 1.7 points, from 58.8% to 57.1%, with 14 of 21 moving down. Individual configurations move by up to 7 points: Muse Code with Muse Spark Open (xhigh) falls 7.0, Kimi K3 in Kimi Code CLI and Codex with GPT-6 Luna (max) 6.2, and Muse Code with Muse Wonder (max) 5.6, while Codex with GPT-6.1 Sol (max) gains 2.1. ➤ Four configurations sit on the cost frontier: Codex with DeepSeek V4.1 Flash (max) scores 47.3 at $0.39 per task, Codex with GPT-6.1 Sol (max) 54.1 at $1.50, Codex with GPT-6 Astra (max) 54.7 at $7.04 and Claude Code with Claude Opus 5.5 (max) 61.7 at $11.42. Costs are pay-per-token API costs, not subscription prices. We will keep updating the Coding Agent Index incrementally, with more evaluations and agents to come.

In AA-PersonalityBench, Gemini models are the most Confident and the least likely to Admit Uncertainty of all models measured ➤ Most Confident: Gemini 3.8 Flash (100), Gemini 4 Argon (97) and Gemini 3.1 Pro Preview (95) ➤ Least likely to Admit Uncertainty: Gemini 3.1 Pro Preview (1), Gemini 4 Argon (4) and Gemini 3.8 Flash (6)

Okay so I managed to squeeze in getting model cards out today for my Muse Wonder Full benchmark with @VulcanBench - enjoy! This, plus all the data from every run will be on the VulcanBench site soon(ish), and all of the traces will be on Github as well. Everything I do with VulcanBench is free and open source. As a truly independent benchmark (it's just me!) I share it all, every single trace from every run, plus the source code for VulcanBench so you can even re-rerun specific tests, or the entire benchmark if you want to. What stands out looking at this is that Astra is fast, but you pay for it. GPT-6.1 Sol is fast and cheap. Opus wins in accuracy, and Muse Wonder is accurate, but expensive and slow. Right now, with the data I have, I don't think I would use Muse Wonder for coding, and I'm not sure that's the market they're going after, yet. That being said, looking at the accuracy numbers, it's clear the Muse team has built a very intelligent model. If they can figure out how to wrangle the time/cost per task, they could very easily become a model usable for daily coding tasks. Live long and benchmark 🖖

Okay my Muse Wonder Full benchmark is now complete with @VulcanBench - busy weekend so might not get to put all the data and model cards together until next week. But I wanted to share a quick peek at the combined score and how it compares to other models. The TL;DR is, with Muse Wonder Full, Medium effort seems to be the sweet spot, but, you need to take tokens-per-task and cost-per-task into account when deciding what model to use. In the two tables below, first is combined accuracy score, second is time per task. Right now Muse Wonder Full is quite a bit slower than Opus 5.5, much slower than GPT-6.1 Sol, and way slower than GPT-6 Astra. Overall, based on what I'm seeing, I would use Opus 5.5 or GPT-6.1 Sol over Muse Full at this point. You'll get the same or better accuracy faster and cheaper. More to come in model cards and deeper analysis on the site, but it's the weekend and I'm hanging with family more than doing benchmarks so might be a day or two. You should do the same! 🖖

Okay, my Muse Wonder Full benchmark with @VulcanBench continues on, but slowly. I upgraded to the $200/mo plan, but still couldn't get through two full effort levels. For comparison, on the $100/mo plan I was able to benchmark Astra on every effort level 👀 That being said, I wanted to thank the team at Meta, they have been jumping in to help, and have acknowledged that there seems to be some issues at lower effort levels that they are looking into. Right now I'm out of usage for a week, but I wanted to share the results I have so far. From an accuracy standpoint, Muse Wonder does pretty well, but it's the time per task where it really runs into issues. But like I said above, this could be an overthinking issue at lower effort levels. In a week or so I should get more usage and can test High and if I still have usage after that, continue up the effort levels. That being said, it is a bit disappointing that I can benchmark any other frontier model in a single sub in a few days without hitting usage issues, and Muse Wonder could take over a month to benchmark given how quickly it runs through usage. Updating my analysis section on the VulcanBench site as well so anyone that wants to dive into the details can. It is a great honor to be able to benchmark the frontier and discover results like this. As I've said many times before, I have incredible respect for the teams building these models, this is crazy hard stuff. I do think Meta is turning a corner here, excited to continue the benchmark when I can 🖖

Insane respect for @alexandr_wang, was not expecting this. Someone from the team already reached out, excited to figure out how to get my Muse Wonder benchmark back on track 🖖

Trying to determine how much it could cost me to benchmark Muse Wonder...every other model I have benchmarked has been able to do it no problem, in a sub. Muse Code's $50/mo tier and can't get through even all the tasks in my benchmark suite at low effort. Looks like I would need a budget between $310-$770 to benchmark Muse Wonder vs. $0 for models from companies like Anthropic and OpenAI since I can run those in my subs. So...trying to think of alternate paths. I've been told that potentially Muse Code just might not be a super optimal harness, and Pi might be a better harness to use it with. Might try this before calling it. Such a bummer because I really like the team at Meta, but I have no horse in this race and have to be independent and just let the data guide me. And right now, I can't possibly understand why I would code with a model that is so much more expensive than the latest models from Anthropic and OpenAI. But, not giving up yet, just can't put $500+ into benchmarking this if I already know I wouldn't recommend it for coding with the early cost-per-task data I'm seeing. Next I'm going to compare a few tasks at low effort in Pi to see if the cost-per-task is lower than in Pi...hoping it's just the harness and not the model 🤞

Out of the gate, I am finding Muse Wonder to be a very expensive model to use. At Low Effort, I hit my weekly usage limit on my Muse Code plan in less than 12 hours. With VulcanBench, I couldn't get through an entire effort level run without hitting my limits. Compare this to even something like Astra, which did every effort level, in under 12 hours, for less than $50. That makes Muse Wonder more expensive, and a lot slower, than Astra. Now I'm not sure what to do. I'll either needs to pay a small fortune to benchmark this model given how token hungry it is, or take a month or longer benchmarking it. Put together some preliminary results comparing Muse Wonder to Opus 5.5 and GPT 6.1 Sol w/respect to token use. While it's looking like an accurate model, I care a lot about cost-per-task, as we all should, and right now, this number is looking very high.