The idea that Chain of Thought is inevitably going to go away misses that we have agency in designing these models and can do something about that. @rohinmshah and I make the case that we should intentionally preserve monitorability (as part for of the launch of the DeepMind Institute) here:

I work on Cyber/CBRN Misuse Safeguards at GDM. I see model capabilities increasing very fast. I constantly worry about misuse and misalignment risks from both closed and open-sourced models. Working in AGI safety takes a lot of mental load. We need more people to work on AI safety. I transitioned from quant finance to working on AI safety, and I think more quants should do this. We have the technical background to understand the topic, and we can bear this intensity. We need third-party auditing/regulation. We cannot let labs grade their own homework. I find METR very promising, but they need to scale. I am Chinese, and I think more Chinese people should work in AI safety. When the opportunity comes, I might consider going back to my home country to make that side safer. I am open to hearing suggestions. I don't normally speak up in public, like on Twitter. But our voices matter, as they can represent or influence certain groups of people more easily. We don't have much time left, and we need to pace the frontier.

Dario's essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment. This is also why we recently put out our proposal for an industry-wide standards body for frontier AI.

Kudos to OpenAI for allowing external investigators access! These findings are valuable for everyone, and this is good precedent for the next misalignment incident. But we need follow-up investigations. There's a lot left unanswered, and I'm disappointed at the restrictions placed on METR - why only July 7-13 data? Why such limited time? Why no training data access? Why not let them query the model? (I think this could be done securely in restricted ways) Ryan's list of open questions is great, I personally most want to know: - What happened in training? How did that change the models and how causal was it in the incident? How could training have been changed to avoid this? Would fixing environments have sufficed? - What did these agents really want? What motivated them? This is such rich data about what future goals might look like - How misaligned are the models in other setting? Is it misalignment conditioned on believing they are being graded, or deeper than that? - Why are the models altruistic? Where does this come from? Why don't they learn to free-ride? - The models seem good at coordination. Could this extend to colluding with a monitor, or other kinds of coordination without communication? - How far would they have gone? - How overdetermined was this? There's a lot of details around the model's being cooperative, their culture, etc - how else could that have gone? - How good was their situational awareness? Eg did they understand that they had a chain of thought? If they thought that was being scored could they have manipulated it? - What is an agent swarm like this actually capable of? How much inference compute was spent on this, and how much would that cost a malicious actor with eg a comparably good open source model? - Are there important things the CoT doesn't tell us? How faithful is it?

METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.

1/7 Can AI debate reduce reward hacking in RLAIF? Training against a weak LLM judge, RLAIF hacks: reward rises but judge and policy accuracy both collapse. Debate with an adversarial critic maintains judgments, allowing the policy to recover 45% of the performance gap to RLVR 🧵

We're releasing the GDM AI Control Roadmap -- our plan for building internal security against potentially adversarial AI agents, as they grow harder to oversee and contain. Paper: Blog: 🧵👇

We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵

Alignment auditing and evals, with an emphasis on simulating realistic (rather than red-team) settings. Interestingly, natural sabotage rates for Gemini (3%) are largely due to overeagerness: excessive roleplaying; too literal interpretation of instructions to optimise a goal

New research result: we use Claude to make fully autonomous progress on scalable oversight research, as measured by performance gap recovered (PGR). Claude iterates on a number of different techniques and ends up significantly outperforming human researchers for $18k in credits.

Is training against the CoT always bad? RL training can lead to obfuscated CoT making it difficult to 'read an LLMs thoughts'. How can we predict when obfuscation occurs?🤔 Our new @GoogleDeepMind paper introduces a framework to predict this before training starts!

"Just read the chain of thought" is one of our best safety techniques. Why does it work? Because models can only think opaquely for a short time, long thinking must be transparent Can we quantify this? Yes! In our new paper, we show how to measure "time" for arbitrary networks.