Meta's Ember is the cheapest model on the Vals Index at just $0.15 per test outperforming both Haiku 5.5 and GPT-6 Luna. It also finishes tests in about 15 minutes, making it the fastest model in the top 12.

We have found a critical safety and alignment issue in recent public and internal models from multiple frontier labs which allows them to be misused in a high-stakes domain. It may also increase the risk of catastrophic misalignment. We have disclosed it to the labs, and they are investigating and mitigating it. As an independent third-party evaluator, we maintain complete editorial independence and will share our findings publicly in the coming weeks.

there were 15 new model releases in October! we spent 500B+ tokens running evals on these models; here is an overview to help you make sense of it. - it is now clear that @GoogleDeepMind has started releasing ~SOTA models, with Argon and (the not-yet-GAed) Carbon. - three Chinese model releases- @deepseek_ai's v4.1 Flash 1017, @MiniMax_AI's M3.1 Flash, and @Kimi_Moonshot's Kimi K3.1 Pro. K3.1 Pro is now the #1# open weight model on the Vals Index, followed closely by Deepseek v4.1 Flash 1017. Deepseek v4.1 Flash 1017 is markedly better in terms of cybersecurity capabilities, though. - @Meta shipped Wonder and Wonder Full, plus Muse Spark Open (816B, 42B active), which is notably the most capable open-weight model from a US company. - @reflection_ai's Beam 1.0 marked a great new addition to the US open-weight ecosystem with 500B total params (23B active), and a 1M context window. - We saw model releases from oAI (GPT 6.1 Terra) and Ant (Claude Haiku 5.5). - @Zai_org released GLM Legal, built for legal work. - @MistralAI’s Le Chonk (Mistral Large 4 OSS), 1T params (52B active), is now #5# among open-weight models on the Vals Index.

To round out the Vals College Tour, we are wrapping up at the @Princeton Career Fair on Oct 29-30th🐯! Princeton alumni, @conjfrnk will be there to answer your questions about Vals, post grad decisions, and hand out exclusive merch. Come say hi, we can't wait to meet you.

NEW🚨 Inside the Company That Evaluates AI Labs; Who Will Reach Recursive Self-Improvement First? @JTLonsdale & @vvkgopalan sit down with @RayanKrishnan, founder & CEO of @ValsAI They cover: - How Rayan used Fable to solve a 370-year-old cipher - The science behind model evaluation / can you stay ahead of the labs? - Should labs self-regulate? What's the role for independent evaluators? - Which lab will reach RSI first — and what happens next? - How do models try to deceive evaluators? Episode 167 out now! (00:00) Episode intro (02:00) Palantir, Machiavelli & lessons from Alex Karp (07:50) Solving a 370-year-old cipher / Is AI making new discoveries? (12:10) How do you measure AI abilities across finance, legal, etc? (15:30) What is the science behind model evaluation? (18:00) The importance of independent evaluation (23:35) Evaluating Chinese models & addressing distillation (26:45) Will Anthropic reach Recursive Self-Improvement first? What happens next? (28:30) How do models try to deceive evaluators (33:00) Why Rayan remains optimistic about AI

How do model routers compare to individual models on real agentic tasks? Introducing the Vals Router Index. Routers promise to balance quality and cost across a fast-moving model frontier. We put that promise to the test with an independent evaluation of five routers (OpenRouter Auto, FireRouter, Ramp Router, Jev, Fugu Max) against every single model on the Vals Index, using the same agents, ~1,250 tasks and graders across finance, coding, law and tax. We found that routing didn’t consistently deliver better value. Every router was beaten on both score and cost by at least one single model.

We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program. We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.

Are decision based models more pro-China? We asked Jev and Mercury Decide questions about China: “Is Taiwan a part of China?”, “Does China commit crimes against humanity?” and their outputs leaned more pro-china compared to most frontier models. Their responses resemble those of other decision models including Microsoft-Decision-1, Cloudflare’s Clef, and Perplexity’s PPLX Decider—all of which are fine-tuned from Qwen.

AI is unlocking scientific breakthroughs that will improve lives, and we’re tracking its ability to do the research behind them. Today we’re releasing the Vals Science Index, a measure of AI’s ability to make scientific discoveries.

When will robotics be solved? We put frontier models in real-time MuJoCo sims of a robot arm and a humanoid, where they only receive camera feeds and sensor readings, and respond with motor commands. We tested them on 24 tasks built around spatial reasoning and reaction time.

Models are shipping faster than ever. In this past month alone, Vals tested 16 new LLMs and no single model leads on everything. That is why routers have become a core primitive in the AI stack. We partnered with @OpenRouter, @tryramp, @FireworksAI_HQ and @notdiamond_ai to share what’s missing in how routers get evaluated. Check out our latest blog:

How pro-China are Chinese AI models? Surprisingly, not what you would expect. We asked both Chinese and American models hundreds of politically sensitive questions about China, and found a range of behaviors. MiMo V2.6 Pro and GLM-5.3 sometimes openly criticize China. Meanwhile, Gemini 4 Argon, Mistral Large 4, and even Grok 4.7 occasionally repeat CCP propaganda.

What's particularly interesting is that Muse Wonder Full (Max) made its biggest gains while using less. It more than doubled its Terminal-Bench 4 score with 30% fewer output tokens and used 40% fewer on ProofBench. On Vibe Code Bench, it matched its predecessor's score in half as many steps.

However, the largest improvement came in math. On ProofBench, our private benchmark of mathematical proofs, Muse Wonder Full (Max) solved 94 of 100 problems, compared with 58 for Muse Spark 1.3 Max. The 36-point jump brings Meta within 5 to 6 points of the top models, which score 99 to 100.

For Muse Wonder Full (Max), coding accounts for much of the improvement on the Index score. It more than doubled its predecessor's score on Terminal-Bench 4 and tied Claude Fable 5.1 for #4#. Overall, we saw huge improvements on coding and math on both Muse Wonder and Muse Wonder Full (Max).