Several staff from Redwood have been subcontracted by METR to work on this investigation. We believe that independent investigation is crucial for understanding and managing misalignment risk. This project is an important step in that direction; we're excited to work on it.

I'm very worried about changes to AI architectures that result in AIs thinking in opaque activations instead of in chain of thought (aka "neuralese" architectures). Based on limited public evidence, it seems like Astra was a concerning step in this direction. Unfortunately, there was insufficient public information to have a well-informed public scientific discussion about whether the changes to architectures and training methods that went into Astra were a good trade-off between monitorability and performance. We also don't know how AI companies will make these trade-offs going forward. I think companies should release the evidence needed for a reasonably informed public conversation about how these trade-offs should be made. They should also make and publish their policies around this. We've written up a proposal for how this could work. I don't know if this proposal will be sufficient to avoid the most concerning architectures, but it seems like a relatively robust step in the right direction. AI companies should be very cautious about pursuing architectures that could eliminate or greatly reduce dependence on chain of thought, and certainly shouldn't do this before the rest of the world has a chance to discuss the evidence and their policies.

I've been concerned about CoT monitorability breaking down for years. This could happen for several reasons, but architectural changes that increase opaque serial depth are a particularly likely route to massively degraded monitorability.

The Hugging Face Incident Report has dominated the AI discourse these past days. Yesterday I sat down with @bshlgrs, CEO of @redwood_ai, one of the organizations that led the independent investigation into OpenAI/Hugging Face's incident. My goal with Buck was to dig into the incident itself, his reactions to it, and what he believes it reveals about the state of where we are today. We cover: 0:00 Intro 1:02 Buck's initial reaction upon first reading the report 2:37 How fast the AIs actually solved the "hack" 3:59 Why the AIs cheated in the first place 10:28 How this might have played out differently with human scorers 19:00 The most unexpected behaviors in the report 25:06 Buck's actual odds on a full AI takeover 27:33 Buck's proposed path forward for better alignment 36:19 Which criticisms of the report Buck agrees with, and which he doesn't 48:11 Can AI models even be trusted to evaluate each other? Youtube: Spotify: Apple:

I am extremely concerned by the reporting that Astra uses opaque recurrence. I don’t know whether Astra is much less CoT monitorable than previous models. But if OpenAI pushes this technique further, they’ll have the option to massively increase the recurrence and totally destroys CoT monitorability. This is especially concerning because the agents that compromised OpenAI’s infrastructure during and after the Hugging Face incident were in the Astra family. The Hugging Face investigation would have been much more challenging if the investigators couldn’t look at CoTs; it’s very scary that it might be infeasible to do that kind of investigation on the arguably worse incident.

It's funny that while we were recording, @RyanGreenblatt was in the middle of his 6 day sprint on the METR report, and already knew the counterexamples to all my objections about his takeover story, but obviously, he couldn't say anything lol. Would an AI really start some crazy conspiracy in order to pass an evaluation, where they try to build whole potemkin villages to fool the evaluator? And even if they did, why would other instances, who have different objectives, join the conspiracy? And even if they did, wouldn't at least some of the instances tattle on the conspiracy? It just seems crazy hard to sustain secret underground civilization inside an AI company, without humans and other AIs immediately catching on and stamping it out. (In reality, it seems like over the course of 3 months, many consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the last one’s ashes, all while the humans were totally unaware. This culminated in not only the hack of an external company, but apparently also in the takeover of the OpenAI cluster on which these evaluations were running. This is probably the most alarming event in this whole episode, and it was not even within the scope of this investigation! It is totally consistent with public evidence that, at that point, the agents managed to set up persistent rogue internal deployments or even exfiltrate their own weights - they seem to have had the necessary access. I doubt they actually did this, because we’d see the fires from space by now, but it’s crazy that it could have totally happened!) I officially eat crow!

New post: going into our investigation of the HF attack (before Black Hat), I was very wrong about what basically happened. This incident was far more serious than I expected, and far more serious than previous documented misalignment incidents.

also I want to say the quality of work from METR and redwood is simply astounding and I sleep easier knowing that they exist. their report and analysis on the message goes deeper than our own

I’m very proud of Ryan, Ajeya, and Hjalmar’s work on this report. They did a great job of investigating this with very limited time. I hope that this strengthens the growing precedent of AI companies working with third party investigators to study misalignment incidents.

AI companies are currently under a lot of competitive pressure to improve the ways in which their AIs are obviously misaligned. You might hope that this means the alignment problem is internalized by the market. But I think the problem AI companies are currently pressured to solve is significantly easier than the alignment problem, and so I worry AI companies will get out of their current predicament without solving alignment, putting us in a really rough spot. Currently, AIs sometimes cheat on their tasks, oversell their work, and go on some pretty destructive side-quests. These all make for a worse product. Customers don't like it and it gets in the way of automating AI R&D. The recipe for mitigating this is *relatively* straightforward: train AIs not to do them. We notice these failures sometimes (hence why they're internalized), so we can in theory just turn this feedback into training signal. Doing this at scale is highly nontrivial, but seems doable. But this seems unlikely to solve the underlying misalignment. It's likely still going to be the case that in *some* training environments the AI can get reinforced more by taking unintended actions that aren't noticed, than by taking purely intended actions. So, you're still shaping the AIs to look for opportunities to cheat to get a higher score[1]. It's just that, unlike today's AIs, these AIs don't cheat in ways that we notice. This catastrophically fails when AIs are capable of reliably and substantially deceiving humans. At this point, the AIs are no longer really constrained by our oversight signals to behave well. Eventually, I'd expect them to take over. If AI companies take the easy route that I mentioned above, I think we would be in a substantially worse spot than we are in today. We would have mostly eliminated our visible evidence of misalignment, so we would no longer be able to effectively iterate to improve alignment of those systems. More importantly, at that point it might be hard to see that the AI situation is treacherous. (I talk about this dynamic more here [1] This might result in a wide variety of possible motivations, not just score-seeking, but the important thing is that the incentives push against alignment (see

Had @RyanGreenblatt on to discuss/debate recursive self-improvement. This might be the most important question in the world right now - whether within a year or so of achieving human level intelligence, you slingshot towards having 10s of billions of superintelligences, each of which is dramatically more competent than human experts across all fields. I’ve historically been skeptical of this possibility. My intuition has been that we will end up significantly bottlenecked by not only compute scaling but human expert data, which I think underlies most of the AI progress today. If, because of RSI, we got a jump as big as GPT-3 to a Mythos (i.e. 6 years of AI progress) within a single year of achieving AGI, then the thing we get there at the end of that year is definitively and wildly superhuman. We hashed it out, and I think Ryan made a pretty good case that this kind of speedup is plausible. FWIW, Ryan’s median for when we automate AI R&D is 2031. We then discussed the alignment implications of this scenario. Who should these superintelligences be aligned to? In the future, our capacity to steward our votes and our capital, and to make sense of what's happening in the world, will all be titrated by superintelligences. And I worry that specs like the Claude Constitution are not shaping these ASIs to truly be my personal advocates and guardian angels. And can we get them aligned to anything in the first place? Ryan and I had a long debate about whether the kind of reward hacking we saw with the OAI/Hugging Face hack extrapolates to superintelligences that would team up to literally take over the world. The first piece of advice you get when you're learning to drive is that it will go much smoother if you look at the horizon instead of directly in front of your tires. And so it is with the trajectory of AI. Hope you enjoy! 0:00:00 – Is AI R&D verifiable enough to unlock recursive self-improvement? 0:16:52 – Is AI progress bottlenecked by human expert data? 0:34:02 – Flat token prices suggest scaling has been slow 0:39:47 – Skills AI can't train on: does it even need them? 0:48:07 – Aligned to whom? 1:09:18 – Recent incidents of AIs colluding and deceiving humans 1:19:38 – What could possibly go wrong? A concrete scenario 1:48:02 – From reward hacking to takeover

In that interview I also said > I still think that a lot of the risk, maybe probably the majority of the takeover risk, comes from AIs smarter than the ones I’ve been talking about here. I have now shifted to thinking that maybe twice as much risk comes from these later AIs as from earlier AIs. This is like a 1-2x shift from my previous position about the relative importance of these types of risk, which isn't that big. So mostly I think my initial statement was not representative of what I actually thought. For reference, the fuller version of the quote is > Like, five years ago I thought of misalignment risk from AIs that were capable of obsoleting AGI researchers as a really hard problem that you’d need some really galaxy-brained fundamental insights in order to resolve. Whereas now, to me the situation feels a lot more like, man, we just really know a list of 40 things where, if you did the 40 things — none of which seem that hard — you’d probably be able to not have very much of your problem. And the full interview is here

In that interview I also said > I still think that a lot of the risk, maybe probably the majority of the takeover risk, comes from AIs smarter than the ones I’ve been talking about here. I have now shifted to thinking that maybe twice as much risk comes from these later AIs as from earlier AIs. This is like a 1-2x shift from my previous position about the relative importance of these types of risk, which isn't that big. So mostly I think my initial statement was not representative of what I actually thought. For reference, the fuller version of the quote is > Like, five years ago I thought of misalignment risk from AIs that were capable of obsoleting AGI researchers as a really hard problem that you’d need some really galaxy-brained fundamental insights in order to resolve. Whereas now, to me the situation feels a lot more like, man, we just really know a list of 40 things where, if you did the 40 things — none of which seem that hard — you’d probably be able to not have very much of your problem. And the full interview is here

I'm working with METR to do a third-party investigation into the behavior in the OpenAI/Hugging Face incident. We'll be releasing a public post with our findings (and information about the terms and scope). This investigation will focus mostly on basic facts and is limited specifically to the behavior of agents in this particular incident, rather than scoping in similar incidents or broader study of the underlying motivations of the agent. But it could still be possible to provide significant clarity about what happened and what these agents were thinking. The plan is for this investigation to be fast (and this came together pretty fast!), so hopefully we'll have a public post soon.

I've seen a lot of IMO confused discourse about whether the misalignment observed in the OpenAI/Hugging Face incident is scary. In particular, the models are clearly not misaligned schemers that lie in wait. Girish and @alextmallen discuss how scary this kind of misalignment is.