Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all. Most of this post analyzes the new METR report (though other sources of evidence are also important). If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario. I am scared, sad, and angry.

I encourage those at frontier AI companies to read this quote, from Ellsberg to Kissinger back in 1968: “Henry, there’s something I would like to tell you, for what it’s worth, something I wish I had been told years ago. You’ve been a consultant for a long time, and you’ve dealt a great deal with top secret information. But you’re about to receive a whole slew of special clearances, maybe fifteen or twenty of them, that are higher than top secret." “I’ve had a number of these myself, and I’ve known other people who have just acquired them, and I have a pretty good sense of what the effects of receiving these clearances are on a person who didn’t previously know they even existed. And the effects of reading the information that they will make available to you." “First, you’ll be exhilarated by some of this new information, and by having it all — so much! incredible! — suddenly available to you. But second, almost as fast, you will feel like a fool for having studied, written, talked about these subjects, criticized and analyzed decisions made by presidents for years without having known of the existence of all this information, which presidents and others had and you didn’t, and which must have influenced their decisions in ways you couldn’t even guess. In particular, you’ll feel foolish for having literally rubbed shoulders for over a decade with some officials and consultants who did have access to all this information you didn’t know about and didn’t know they had, and you’ll be stunned that they kept that secret from you so well." “You will feel like a fool, and that will last for about two weeks. Then, after you’ve started reading all this daily intelligence input and become used to using what amounts to whole libraries of hidden information, which is much more closely held than mere top secret data, you will forget there ever was a time when you didn’t have it, and you’ll be aware only of the fact that you have it now and most others don’t….and that all those other people are fools." “Over a longer period of time — not too long, but a matter of two or three years — you’ll eventually become aware of the limitations of this information. There is a great deal that it doesn’t tell you, it’s often inaccurate, and it can lead you astray just as much as the New York Times can. But that takes a while to learn." “In the meantime it will have become very hard for you to learn from anybody who doesn’t have these clearances. Because you’ll be thinking as you listen to them: ‘What would this man be telling me if he knew what I know? Would he be giving me the same advice, or would it totally change his predictions and recommendations?’ And that mental exercise is so torturous that after a while you give it up and just stop listening. I’ve seen this with my superiors, my colleagues….and with myself." “You will deal with a person who doesn’t have those clearances only from the point of view of what you want him to believe and what impression you want him to go away with, since you’ll have to lie carefully to him about what you know. In effect, you will have to manipulate him. You’ll give up trying to assess what he has to say. The danger is, you’ll become something like a moron. You’ll become incapable of learning from most people in the world, no matter how much experience they may have in their particular areas that may be much greater than yours.” "….Kissinger hadn’t interrupted this long warning. As I’ve said, he could be a good listener, and he listened soberly. He seemed to understand that it was heartfelt, and he didn’t take it as patronizing, as I’d feared. But I knew it was too soon for him to appreciate fully what I was saying. He didn’t have the clearances yet."

I'm just speculating like everyone else (no juicy secrets about this shared with me) but here's my guess about what is going on right now with OpenAI and the three fired employees: The negotiations between OpenAI and METR are ongoing. OpenAI has said they want embedded third parties, METR has proven their worth with the previous engagements, so METR is an obvious choice. But my guess is that METR is asking for more this time; probably they want more access, more time on site, less narrow scope, the right to comment on redactions, etc. (Compare with the terms of their previous engagements: In this one a third of their report was redacted and they weren’t allowed to say why, and in the Hugging Face investigation they only had 6 days and were only allowed to investigate the precursor swarms nor the subsequent more severe compromise of OpenAI infrastructure. Various researchers at OpenAI probably support METR in this (including, I’d imagine, the fired three employees? Tomek was the point person on the hugging face investigation), but also various higher-ups at OpenAI and legal types probably oppose and want more restricted access and scope – basically they want to work with an org that’ll go easy on them and not give them trouble. OpenAI holds most of the cards here, since this whole thing is voluntary and not legally required, and they can threaten to go with Accenture or HuggingFace or whatever instead. Anyhow to improve their bargaining position and/or minimize the blowback if they decide to say no to METR and go with someone more pliable, it helps if they can discredit METR. Hence (I suspect) why they are going after my friend Ryan Greenblatt (who works at METR) now. They don't need to present actual evidence, they just need to cast vague aspersions, and many employees will give them the benefit of the doubt. If and when they say "We chose to work with a different third party" they'll face lower risk of employee discontent than if METR's reputation was unsullied. This whole mess, by the way, was predictable. *Of course* there's a big political fight about who should be trusted to do this and who has relevant expertise and independence etc. and *of course* the companies are trying to make things shake out in a way that favors them. It's part of why I've been advocating for *actually pacing the frontier* instead of just doing third party assessments. "We allocate at most 20% of our compute to AI R&D" is a lot easier to verify than "we are being safe and responsible" and depends a lot less on subjective judgment and technical alignment expertise -- AND, if you do it, people will be able to see the slowdown in the trends of agentic coding performance in the frontier companies and see laggard companies start to catch up. Harder for it to end up just being safetywashing / industry capture.

Helpful list of all the recent rogue AI incidents from the WSJ. It's getting hard to track them and will only get worse. We like need to establish consistent naming or numbering conventions, e.g. OpenAI-May11June26-Collusion.

oh man i’m sure that accenture will have lots of expertise at, uh, *checks notes* redteaming models, assessing alignment, and testing safeguards. after all, their website talks about embracing the power of change to create 360° value and shared success for clients, people, shareh

I agree with the vice president here. These companies should just be stopping, citing the danger. Then they'd have much more credibility when it comes to communicating that others must be stopped too, domestically and abroad.

214 million people saw this AI warning. So we called an emergency debate. The warning came from someone who had worked at both Anthropic and OpenAI. Then a current Anthropic employee backed it publicly. It had spread so far beyond the tech world that a friend of mine who cuts hair and has never really cared about AI messaged me asking, “What the hell is going on?” I then realised a lot of people were probably asking the same question. The problem is, when you speak to people who have spent years studying AI, you get completely opposing answers. So I brought four of them around the same table. Roman Yampolskiy is a computer scientist who coined the term “AI safety” and has spent years studying whether increasingly intelligent systems can remain under human control. Nate Soares leads the Machine Intelligence Research Institute and has spent more than a decade working on AI alignment. He believes we are moving too quickly towards systems we don’t yet know how to reliably control. Ed Zitron thinks much of the AI conversation has become detached from what the technology can actually do today. He believes the industry is overhyping it while distracting us from financial, environmental and social consequences already happening. Andrew McAfee is an MIT researcher and economist who takes a very different view. He thinks we spend so much time talking about what could go wrong that we barely talk about what AI could make better. And that disagreement is what made this conversation so interesting to me. We discussed things like: - How do you control something that eventually becomes smarter than you? - Are the biggest warnings about AI based on evidence or assumptions? - What happens to work and human purpose if AI becomes better at more cognitive tasks? - Are we ignoring problems AI is already creating because we’re obsessed with hypothetical future ones? - Why have Sam Altman, Elon Musk and Geoffrey Hinton all warned us about AI? The question I kept coming back to was simple: What is actually true? Depending on who you listen to, AI is either one of the greatest opportunities humanity has ever created or something we’re racing towards without understanding the consequences. Both claims deserve to be challenged. There were moments in this debate where I genuinely found myself moving between the arguments. That’s the value of putting people who fundamentally disagree in the same room. I didn’t want four people telling me the same thing. I wanted each of them to explain where the other side was wrong. If you’ve watched the last few months of AI news wondering what you’re actually supposed to believe, this conversation is for you. Our emergency AI debate with Ed, Roman, Nate and Andrew is out now ❤️👊🏾

Screenshots of the Philip Morris website. (reminder: they're a leading cigarette company) It's a good calibration point for how good modern marketing is at dressing up pretty much any corporate (or, for that matter, government) behavior and making it sound responsible and safe.

It’s really great that several top AI labs have said they will develop safety cases and work more closely with external assessors. This has been at the top of my wish list for awhile!! But I still worry a bit about how this might go in practice (this is a concern about the field in general, not about OpenAI specifically): • By default, I don’t think alignment and control cases will support a quantitative measurement of absolute risk, e.g. “catastrophic risk from covered activities is <1% over the next 3 months.” • Ultimately, making such a measurement depends on an argument about generalization from alignment or monitorability evals to deployment. • I don’t think we understand generalization well enough to make scientific claims like this. (If we did, I think we’d have basically solved alignment.) This could lead to a situation where an external assessor says “we can’t convincingly rule out low or high risk,” and incentives push towards anchoring on “lack of evidence that risk is high” over “lack of evidence that risk is low.” What could improve the field’s epistemics here? A couple ideas I like: • Have a committee of ~10 people deeply read a risk report and give their own subjective probabilities of risk, then report the distribution or median. • Have ~3 people involved in writing the report each contribute a short, signed appendix with their own subjective probabilities and the arguments behind them. In either case, previous reports and estimates could be provided as context, so there is at least an attempt to accurately capture relative risk to prevent frog-boiling. I think it could be valuable for labs to at least start trialing this internally for high-profile safety cases or risk reports. Publishing these assessments could be even better, but I can see that being challenging for various reasons, and even going through the exercise privately seems like it could be quite valuable. Very curious what other ideas people have for improving epistemics around risks (including ideas for better science around generalization).

This is crazy. In late July, "three guys with Claude and Codex subscriptions" were able to use Opus 5 to access OAI auth tokens and gain write access to OpenAI's monorepo openai/openai over the course of two days.

The last week has really shown me that someone who wants to understand AI risk has no good place to start. Hence we made the Wirecutter for content about AI Risk. We're launching with 3 articles: 🧵

I’m very concerned that during RSI, labs will just stop externally deploying their models. Which means they'll be going full steam ahead on the most dangerous use case of these models (recursive self-improvement), while the public remains in the dark about the nature of capabilities and the state of alignment. And we end up on a path towards tremendous concentration of power.

Another good essay from an OpenAI capabilities researcher I know (well, now former). I think focusing on control and abandoning alignment for now is is a reasonable strategy, but implementing it effectively will require an agreement not to RSI to ASI at max or close to max speed. Figuring out control is going to take time and money. Also, it only works up to a point, I think. In AI 2040 we talk about this.