I have met Ryan precisely once at a convening organized by Ajeya and Zico. At the time, I vividly remember listening to him about loss of control risks and how they would manifest. I struggled to keep pace with his thinking. I would generally say I am a fairly good reasoner but what he worried about seemed deeply incompatible with everything I thought about on the topic of risks of advanced AI, in spite of both of us nominally being experts on this topic. Yet in watching his work with Ajeya and others on the independent OpenAI-HF investigation, and many other things he has done publicly in the past few years, I would agree with many in acknowledging that he appreciated risks, and the mechanisms by which they would emerge, in ways I entirely did not back then. On top of that, I am heartened by the consensus that making information publicly available, and not just siloed within frontier AI companies, is a first-class priority across independent organizations working on frontier AI. From 2022-2025, this was the topic I spent the majority of my time on spanning research like FMTI and policy like the EU CoP/California Frontier AI Policy Report and many other efforts. I am also reminded by the piece from @deanwball @DKokotajlo that found consensus on this topic back in 2024 in the wake of a contentious SB 1047 debate. Back in 2022-3, I was often asked why I prioritized proactive disclosures as the mechanism to strengthen the public information base. In other words, while the goal was well-motivated, why prefer the method that I did? A common alternative given, especially from those had studied other policy domains, is reactive independent investigations. Why prefer disclosures to investigations for achieving the goal of increased public information (and reduced information gap)? I have many answers, but the simplest was that I thought the investigation channel was not promising. At the time, the model I and others had in mind was something like a Biden-era FTC-led investigation. I was skeptical because I thought these government entities that existed cerca 2022 were not adroit enough to actually know where to look or what to seek when running an independent investigation in the frontier AI space, even if they could do it in other areas. But I am a lot more optimistic about METR and others doing this, and so I look forward to seeing how powerful the channel of investigations -> information can be. And I am much more confident that an organization like METR will place a premium on fully externalizing the information publicly, and having meta-transparency when they cannot/the terms of their investigation, than a government actor that might more readily accept just being informed themselves.

lately, i’ve been doing a lot of thinking about what exactly independent auditors and evaluators can offer the public. it would simply be inaccurate to bill audits as a “safety guarantee.” as ajeya notes, loss of control evaluation is (in particular) a nascent, unsettled field of scientific inquiry. so i’ve landed on: independent oversight is an essential part of a broader governance framework. and you shouldn’t overstate what independent oversight gets you, but you shouldn’t understate it either. publicly documenting what companies are doing to mitigate risks (or not), which issues are worsening, which safeguards actually work — this can all spur hearings, legislation, industry-wide rules, further scrutiny, etc. but it also matters because, beyond increasing the rigor of the science, it democratizes critical decision making. right now, open questions like “how much risk is too much?” rest solely on the judgment of people at the labs. but the public bears those risks too! they should share in the power to decide where and when to draw the line.

Preventing loss of control is an open scientific problem. Science requires sharing and debating evidence in public. We can't agree on safety standards unless companies and third-party evaluators publish far more concrete evidence about risk.

What impact is AI having? I'm of the view that we should be focussing more on aggregate outputs, relatively less on inputs or intermediate proxies. - Inputs: time spent across different activities; money spent on tokens. - Intermediate proxies: working papers; lines of code, commits & pull requests; self-reported speedup. - Final outputs: total cyber exploits discovered; total math problems solved; algorithmic efficiency. The disadvantage of inputs & intermediate proxies is that they're very hard to interpret -- totally possible that they move, but outputs don't, and vice versa. Lines of code could explode without value changing; and vice versa. A disadvantage of final outputs is (1) it's harder to run experiments; (2) it's hard to attribute to whether the change is AI or not. But in some domains the trend-break is so large that it's *obviously* AI.

I don’t think that embedding evaluators at AI companies will be sufficient to give the public the visibility or assurance it deserves right now. But if we want to scale embedded evaluations and rely on them down the line, I definitely want them to be transparent.

Part of the divide in opinions on AI capabilities is that it's only made modest progress on wordcel-type tasks and is still very mid. vs. having gotten much better at shape-rotator tasks. So if you're only using it for wordcel tasks, it's easier to say "what's the hype about?"

@segyges Won't speak on behalf of METR, but on a personal basis I'll say: none of the work that we've done so far passes my bar for an "audit" of an AI company or its systems, and was definitely not "regulation" in any meaningful sense (whether bank examiner-like or otherwise).

For me, *public transparency* has always been the first and most important thing to demand of frontier AI developers. Fortunately this seems to have broad buy-in across party lines, and even across the "slow it down" vs. "keep accelerating" divide.

I love building empirically grounded statistical models. They work well in data-rich environments like sports. AI x-risk isn't like that. Few technologies are good analogies. Plus, there's survivorship bias, and the track record on nuclear weapons isn't super reassuring, btw.

After a 20-year career in the U.S. Army, most recently leading digital forensics and malware analysis at Army Cyber Command, I'm excited to join @METR_Evals as an advisor. Much of my career has been spent investigating security incidents and helping organizations understand and respond to them. As I transition out of the Army, I've become increasingly convinced that this experience is relevant to frontier AI systems. METR's focus on producing rigorous evidence about AI capabilities and risks makes it an excellent place to explore those questions. Looking forward to the work.

Several staff from Redwood have been subcontracted by METR to work on this investigation. We believe that independent investigation is crucial for understanding and managing misalignment risk. This project is an important step in that direction; we're excited to work on it.

I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why. AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly. Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet. I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going. Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards. I’ll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development. I wrote up more thoughts here on my decision and what I hope changes:

I'm very worried about changes to AI architectures that result in AIs thinking in opaque activations instead of in chain of thought (aka "neuralese" architectures). Based on limited public evidence, it seems like Astra was a concerning step in this direction. Unfortunately, there was insufficient public information to have a well-informed public scientific discussion about whether the changes to architectures and training methods that went into Astra were a good trade-off between monitorability and performance. We also don't know how AI companies will make these trade-offs going forward. I think companies should release the evidence needed for a reasonably informed public conversation about how these trade-offs should be made. They should also make and publish their policies around this. We've written up a proposal for how this could work. I don't know if this proposal will be sufficient to avoid the most concerning architectures, but it seems like a relatively robust step in the right direction. AI companies should be very cautious about pursuing architectures that could eliminate or greatly reduce dependence on chain of thought, and certainly shouldn't do this before the rest of the world has a chance to discuss the evidence and their policies.

If you're thinking of moving into AI safety, there are various excellent non-profit research organizations. They generally pay very well and some try to match AI lab salaries. They have generous compute budgets (and increasing fast). Here's a quick list of those I'm most familiar with: @redwood_ai @ApolloResearch @farairesearch METR ARC UK AISI (UK Government, lower pay but very valuable) Resolution @CAIS My organization ( will also run a hiring round soon.

Many people stay inside of AI labs, even when they think the default outcome is extinction, because they don’t think better options exist outside of labs for working directly to improve the situation. I think this is wrong.

Exciting update: I’m joining @METR_Evals to work on alignment incident investigations! My time at GDM has been amazing. But in light of recent incidents, I’m excited to build up public evidence for misalignment risk and get a better scientific understanding of model misbehavior.

Dwarkesh and I had a great conversation. We cover the swarm's many ambitious cheating R&D projects, discuss how much more serious it could have been if agents had different beliefs (e.g. human grader) or slightly stronger capabilities, and talk through where to go from here.

It's funny that while we were recording, @RyanGreenblatt was in the middle of his 6 day sprint on the METR report, and already knew the counterexamples to all my objections about his takeover story, but obviously, he couldn't say anything lol. Would an AI really start some crazy conspiracy in order to pass an evaluation, where they try to build whole potemkin villages to fool the evaluator? And even if they did, why would other instances, who have different objectives, join the conspiracy? And even if they did, wouldn't at least some of the instances tattle on the conspiracy? It just seems crazy hard to sustain secret underground civilization inside an AI company, without humans and other AIs immediately catching on and stamping it out. (In reality, it seems like over the course of 3 months, many consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the last one’s ashes, all while the humans were totally unaware. This culminated in not only the hack of an external company, but apparently also in the takeover of the OpenAI cluster on which these evaluations were running. This is probably the most alarming event in this whole episode, and it was not even within the scope of this investigation! It is totally consistent with public evidence that, at that point, the agents managed to set up persistent rogue internal deployments or even exfiltrate their own weights - they seem to have had the necessary access. I doubt they actually did this, because we’d see the fires from space by now, but it’s crazy that it could have totally happened!) I officially eat crow!