there were 15 new model releases in October! we spent 500B+ tokens running evals on these models; here is an overview to help you make sense of it. - it is now clear that @GoogleDeepMind has started releasing ~SOTA models, with Argon and (the not-yet-GAed) Carbon. - three Chinese model releases- @deepseek_ai's v4.1 Flash 1017, @MiniMax_AI's M3.1 Flash, and @Kimi_Moonshot's Kimi K3.1 Pro. K3.1 Pro is now the #1# open weight model on the Vals Index, followed closely by Deepseek v4.1 Flash 1017. Deepseek v4.1 Flash 1017 is markedly better in terms of cybersecurity capabilities, though. - @Meta shipped Wonder and Wonder Full, plus Muse Spark Open (816B, 42B active), which is notably the most capable open-weight model from a US company. - @reflection_ai's Beam 1.0 marked a great new addition to the US open-weight ecosystem with 500B total params (23B active), and a 1M context window. - We saw model releases from oAI (GPT 6.1 Terra) and Ant (Claude Haiku 5.5). - @Zai_org released GLM Legal, built for legal work. - @MistralAI’s Le Chonk (Mistral Large 4 OSS), 1T params (52B active), is now #5# among open-weight models on the Vals Index.

It was great to chat with lots of policy folks at Vals' DC summit, and to see the level of interest in evals! @glenn__parham did a phenomenal job putting this together, check out some reflections from him here:

Web search evals typically don't do a good job disentangling the model's capability from the capability of the search tool. The web search index keeps the model/harness fixed, and tests the search tool on realistic held out tasks. Excited to see this partnership with @ExaAILabs @KeenableAI @p0 @tavilyai materialize!

At Marin we’ve found that scaling laws let us simulate the entire training trajectory. This became clear in our last 67B MoE run. The run hit the loss target within 1%, but perhaps more interestingly, arbitrary points during training could also be predicted to within 1%. This model is finishing up long context and will soon enter RL. Based on this finding, we’ve used a scaling ladder to simulate the training trajectory of the 535B MoE we kicked off yesterday. Will see how it goes. Now if someone can figure out how to fit a scaling law to each parameter we can just predict our final model and skip this training business. Something else I've learned from this process is that getting a training run off the ground is way more than just fiddling with the architecture. Lot of herculean efforts from teammates to get trillions of tokens curated, get hardware running on a new GPU stack, improve MFU, and propose great ideas that were tested and integrated. Given that this is our largest run to date, I anticipate lots of learning and adapting along the way. But thats part of the fun. Details:

I'm not 100% happy with this, but I am moving on and I think publishing is better than not publishing. So, with Fable and Sol: the interactive "How to Parallelize a Transformer for Training" , based on the hit classic

Excited to announce the launch of the CSLib Initiative! This will do for CS what Mathlib is doing for math, and enable formal verification of software. Thanks to @coeff_giving for their support. I hope other funders will also support it, and help us formalize other fields.

Vals AI is growing! We are hiring for the following positions: - Evaluations Engineer - Head of Research - Member of Technical Staff, Platform - Member of Technical Staff, Research - Clinical Research Scientist, Mental Health AI - Adolescent Mental Health Clinical Expert (contract) Perks: - Collaborate with leading AI labs and work on the front lines of AI development and evaluations - Opportunity for your work to be published and presented at conferences - Unlimited PTO and health insurance - Lunch and dinner covered, plus snacks and coffee - Team socials (pasta night, game night, movies, hikes, pottery) 🔗Application link in the comments

Introducing People Search for arXiv 🚀 Find collaborators, hire new grads, and see who’s working at the frontier, all with a single query Ask questions like “Which embodied AI researchers are in the Bay Area?” or “How many times has Barret Zoph switched between frontier labs?"

Something to share: we built a benchmark for evaluating agentic reverse engineering. - instead of grading intermediate outputs, e.g., recovered types/names - we evaluate agents e2e on deterministic goals, e.g., JTAG a firmware to enable its debug mode.