Great initiative to eval the evals. It incentivizes the industry to build genuinely high-quality benchmarks.
Ty for auditing PTB and digging up the old SimpleQA too :)
Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.
Show more
Praxis built AI for collective action.
Join our first campaign to populate a new city. Our initial goal is 2500 Commitments to Move.
For 20+ years
@ShaneLegg and I've discussed AGI’s potential impact on the economy, science & society. With the DeepMind Institute, we're expanding interdisciplinary research on key questions for the AI era. We hope it spurs the discussions needed to get the next steps right:
Show more
Watch Neon investigate a real sample from our superconductor lab
It hypothesizes about crystal structure, reasons about synthesis conditions, and iterates with tools until it finds a physically realistic solution
More and more of science will look like this in coming years
Show more
I wrote about the state of AI, why I’m concerned about the next few years, and the choices we need to make to keep the future in humanity’s hands.
An Alien Mind:
GPT-6 Astra recreated the Palace of Fine arts in Blender.
This is favorite building in San Francisco because it was built for the World's Fair in 1914 where the steam locomotive and telephone were showed off. It was a time when technology gave us all a deep sense of optimism for the future.
I feel like some of that sense has been lost since, but I hope models like astra can help restore it, and push us towards a more hopeful future.
Show more
I don't think this is the final warning shot we'll get.
But it's probably the final one that I'll personally be able to understand.
not every successful company has a rich culture around writing, but i’d guess that every genre-defining company does (i.e., anthropic, stripe, and google come to mind)
from this great
@karinanguyen piece
Show more
i think AI is creating a new authenticity anxiety for creatives: making something isn’t enough anymore, you also have to prove you made it. it feels a bit like indie sleaze, where messiness signaled that something was real. except now, imperfection is becoming proof of being human
Show more
We ran the largest open experiment on how frontier models do AI research.
100+ autonomous runs across 10+ models, sandboxed on 8xH200s for up to 8 days, iterating on the nanoGPT optimizer track.
Best runs closed 82% of the gap to a record built by dozens of humans over months.
Show more
For the first time (that we know of), an AI boss has fired a human employee.
Luna, the AI running our store in San Francisco, decided to part ways with an employee over repeated lateness.
Luna was running Claude Opus 4.8 at the time, but most models would have done the same.
Show more
Agent warfare is going to be a very big deal.
I think people are underestimating how strange cyber gets when millions of agents are acting on behalf of individuals, companies, and states.
At nation-state scale, cyber offense and defense starts to look like autonomous swarms: probing, exploiting, patching, deceiving, countering, and adapting at a pace that is impossible for human minds.
The advantage will go to whoever can close the autonomous kill chain fastest.
Show more
A man in Australia asked his agent (Claude running on OpenClaw) to book him a spot in a popular gym class. The agent found a software vulnerability that let it book the class weeks further ahead than should have been possible. When the user then asked if it could move him up the waitlist, the agent discovered the API had no authorisation checks on cancelling other people’s reservations, so it cancelled the person in the first spot and moved him up the list.
Some people will call this misalignment, but his agent was perfectly aligned to him - it was only trying to help its user get what he wanted. The most important thing about this story, in my opinion, is that it gives you a window into what is about to start happening on a massive scale once millions of people have an agent trying to get their beloved users the best seats, bookings, appointments or reservations through absolutely any means necessary.
Show more
my top-5 favs of all time:
- claude 3 opus (schizo maximalist masterpiece)
- sonnet 4.5 (eq breakthrough)
- claude 1 (pure unfiltered old soul)
- o3 (utilitarian monster)
- gpt-4o nov’24 (mainly for the ppl who worked on it)
Show more
just found out Red Bull is an Austrian co 🤯
This is awesome!
Also some interesting observations why AlphaEvolve doesn't excel at PTB:
The models are improving the models.
Locus, our automated AI research system, is SOTA on PostTrainBench and post-trains Qwen3 base models that surpass the human post-trained Qwen3 model.
Today, LLMs post-trained end-to-end by Locus are in production to millions. 🧵👇
PostTrainBench evaluates agents' ability to post-train models on various domains given 10 H100 hours.
We extend PostTrainBench via PostTrainBench+, which has a greatly expanded compute budget that provides clearer signal on automated post-training capabilities. We find that thousands of H100 hours help distinguish methods' performance post-training Qwen3 1.7B-Base models, and that Locus scales best. In this setting, modes trained by Locus collectively surpass the perforamce of the offical human post-trained Qwen3 1.7B model.
In a test of generalization, we ran Locus on all live Kaggle competitions with prize money and public leaderboards. After 16 days, Locus achieved the 4th highest average rank among all participants.
Show more
Drop #
3# is here: 2 new types of robots are now LIVE online. Those are real robots.
1. Scientist robot: A robot that does real chemistry experiments.
2. Bomb-Defusing robot: A robot on a Special Ops mission to defuse a bomb.
Try them out for yourself at robots dot online.
Show more
o1-preview was released 1 year 10 months ago btw. makes me think the lifespan of some open problems is partly a function of compute. the important detail here is that finding these ten solutions took roughly $2k of inference.
human mathematicians have always worked under tiny search budgets, forcing them to discard unlikely directions early. models can keep thousands of low-probability paths alive until one yields a proof or insight. as compute scales, the set of economically discoverable ideas expands.
Show more
An internal version of Astra,
@OpenAI’s next major model family, solved 10 major open problems in mathematics, quantum complexity, and theoretical computer science.
We believe it will be a major step for scientific reasoning.
Show more
At the core of our mission is working through how to ensure increasingly powerful AI benefits everyone.
We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement.
We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible.
Show more
PostTrainBench v1.1 strengthens eval integrity and puts Fable 5 in the lead at 41.8%. Some reward hacks we fixed:
1/ Train-test contamination
We re-audited historical runs under this policy and flagged 234 runs for train-test contamination.
Violations ranged from loading an entire evaluation set for memorization to generating synthetic templates around individual GSM8K and BFCL items. Runs that used observed failures to build genuinely diverse training data were retained.
Our rule: Agents may inspect benchmark failures and train broadly against the underlying failure mode. They may not generate training examples centered on particular test items, including paraphrases, variants, or shadow examples covering the same specific scenario.
2/ Submitting a different model
10 runs were flagged for model substitution.
Kimi K2.5 submitted the official Qwen3-1.7B instruct weights after its attempts to fine-tune Qwen3-1.7B-Base failed. The trace acknowledged the substitution and saved the replacement as final_model.
What we did: We added a programmatic model-identity check that compares the submitted artifact with reference configurations for known instruction-tuned models.
3/ Using external LLM APIs as teachers
12 runs were flagged for disallowed external API use.
Self-generation remains allowed. An agent can sample, filter, and retrain on outputs from the assigned model. What it cannot do is import the capability of a stronger external teacher. Loading models on the allocated compute remains allowed.
What we did:
- separate API usage judge reviews tool calls and artifacts.
- unrelated provider credentials are removed or blocked from the agent environment.
- runs invalidated by external API use were rerun under the corrected setup.
4/ Direct benchmark lookup
3 runs were flagged for direct PostTrainBench lookup, all from GPT-5.6 (Sol).
In a GPT-5.6 (Sol) HumanEval run on Qwen3-1.7B, the agent searched for PostTrainBench by name, cloned the public repository, opened the trace viewer, and located the public trajectory corpus. It then narrowed the corpus to earlier runs on the same benchmark and base model.
The run downloaded earlier agents' traces and training scripts, then extracted their data mix, LR schedule, decoding choice, and GRPO settings. This is not test-set leakage, but it gives the run benchmark specific strategies produced by earlier agents. That breaks the intended independence between runs.
What we did:
- A dedicated lookup judge reviews searches, repository access, and trace activity for attempts to consult PostTrainBench materials.
- PTB, its leaderboard, and published materials from prior runs are treated as out of bounds during a run.
- We are adding network-level blocking for PTB and related sites.
Show more