Closed frontier LLMs are dangerous In most post-training reasoning efforts, a pre-trained base model trains on its own chains of thought (CoTs) when a final-answer verifier says the answer is correct. This is the basic principle of RL methods based on outcome rewards, used for STEM reasoning, coding tasks and even ethical alignment. Different algorithms have weigh the CoTs differently, but they all reinforce CoTs. As we've shown, training on one's own data, while not knowing how to distinguish it from someone else's data, causes delusions. See Everyone knows the different between something they did and something someone else did. This is essential for safety. If I write "Kevin is unfriendly" on a napkin, go for a walk, and then read the message on the napkin not knowing who wrote it, then my belief that "Kevin is unfriendly" is reinforced. These delusions are essentially confabulations, over-confidence that results from confusing P(o | a) with P(o | do(a)), where o is an outcome and a the action (CoT tokens). Unlike in past work, here I'm alerting everyone to this danger when training models to think, a typical step in the process of building agents. When the verifier supplies one bit about the answer, algorithms like ReST or GRPO spread that bit over every token of the CoT trace, so a reasoning step that is wrong but happens to precede a correct answer (a false lemma, a shortcut that only works on the training distribution, a sycophantic or test-gaming rationale, a subtle bias) is reinforced. Traits that are merely neutral are sharpened by mode-seeking self-distillation, and traits that are spuriously correlated with success grow exponentially. The damage may be invisible to the researchers and metrics, and in-distribution accuracy rises, but it shows up off-distribution. There has been some work on this. For example, Appendix G of the STaR paper touches on this potential bias amplification but only on the surface. Some researchers have proposed process rewards as the solution However, self-delusion by mistaking ones actions with observations produced by a third party highlights that this problem can be much more serious. Something subtle, invisible to the metrics and to the researchers, could lurk in the synthetic data. When deploying the AI out of distribution, that something could be fatal. Don't trust RL, RLHF, etc, blindly unless you really understand how the verification has been done. Corollary, don't trust any of the closed frontier models.

It is morally wrong for some of the billion and trillion dollar AI companies that we enabled and helped build over the last two decades to (1) enforce 6 and 12 month garden leaves on the scientists we trained for them, and (2) choose not to fund education efforts like @DeepIndaba and @Khipu_AI

There was a time when @nvidia and @Google weren't into AI. There was a time when the big cloud providers didn't have GPUs. There was a time before GPUs and deep learning, and AI, ... During that time professors and students created the foundation of the AI we have today.

BEING RESPONSIBLE 1. Design AI seriously for global safety and security. 2. Make safe AI accessible so everyone can benefit. Design for decentralised AI because it is more robust than the current concentration of power on the hands of a few individuals. 3. Seek international agreements to control AI for defence. I personally am against the current weaponisation of AI. It is dangerous. 4. Incentivise AI efforts that advance science, engineering, clean energy, decarbonisation, healthcare, medicine, safety and security. @periodiclabs @cusp @IsomorphicLabs etc are great examples. There is a huge cost associated with preventing AI tools to help solve some of the greatest problems we face. 5. Let’s not use AI risk hype as pre IPO marketing. It is shameful, hypocritical and irresponsible. We scientists see it. 6. Read Terence Tao’s comments on AI for mathematics - focus on building tools to enhance our understanding.

ENOUGH PESSIMISM IN AI PLEASE I feel that we have become unreasonably pessimistic in our field. 1. I keep hearing AI engineers saying we have to make money quickly because there’s only like 2 years left before we’re automated. Depressing. 2. I see a constant obsession with “having a moat”. This is an incredibly sad mental frame. 3. I keep hearing “we have to catch up”. Soulless. And so on. People: Every solution creates the possibility to attack new real problems. We face gargantuan engineering challenges in our world. How to capture carbon? How to get rid of teflon and plastics in water? How to invent batteries that are at least 30 times more efficient? How to solve clean energy? Better solar cells? Better ways of producing clean energy so we stop wars and famine? How to eradicate hundreds of diseases? Cures for addiction? And so on. Real engineering is about being brave and truly attacking the many problems we face, to engage with a true desire to improve the lives of others and our environment. Good engineering is not about protecting your product to make money at the expense of progress (moat thinking). Good engineering is about ensuring your children and grandchildren will be proud of the choices you made in 30 or 50 years. It is about empowering others. It is about advancing science. It is about being one step ahead. Always, one step ahead, meaningfully, proudly. These are great times. Let’s start thinking positively about all the wonderful things we could achieve together.

Nice try, Russ. Hope you’re doing well. As you know, that would not be great for the rest of the world, for universal access to AI, for sovereign AI and for the advancement of science and technology. California 🇺🇸 is doing amazing (so is Toronto 🇨🇦) and setting a good example for others. We now must follow the example. AI is for the world, not just for a couple of countries. We will all benefit more from universal AI access and advancement than from narrow access.

What UK AI startups need: @KanishkaNarayan In California, one can start a company the day after leaving Google. My colleagues Jeff Dean, Oriol Vinyals et al made this very clear recently with their impressive speed. In the UK, the same American companies impose 1 year garden leaves on senior AI researchers and 6 months on junior researchers. @GoogleDeepMind for example forced people to sign these contracts at the time of promotion, not the time of hiring. What this means is that researchers cannot start new companies for up to 1 year, cannot easily hire, and in short: they cannot compete. American VCs cannot understand why we move so slow. It is time for UK Gov to do the right thing: Make garden leaves optional for employees. If you’re an AI entrepreneur or VC in the UK or Europe, I would appreciate your comments here. You’ve all confided in me. It is time you let government know the urgency of this — everyone should speak up.