So not to go all Yudkowsky Monocause Theory. But a lot of AI company talk that *sounds smart* did come from me, or me via Bostrom or other diffusion modes. High-powered ideas like "recursive self-improvement" and "AGI" are ideas they did not invent themselves, but got off higher-quality outside thinkers. There's a big intellectual history behind me in turn, but the AI companies did not study this history independently; they got it downstream of my writing it up. I naturally spent some time trying to figure out what the positive vision for AGI and ASI would look like, back in the mid-2000s. But I considered it a lower priority to write up. And AI company executives are not at the intellectual level where they can invent coherent answers to questions like that, if they don't have my answer sheet to copy from. They can't tell you what the positive vision for AGI or ASI looks like, because I didn't prioritize writing up that part of my earlier thoughts. They can't copy it off from the same answer sheet they used to produce smart-sounding conversation like "general intelligence" or "recursive self-improvement" or "extinction risk" or "AI alignment problem". If they're not reading off my answer sheet, AI company executives are pretty ordinary as intellectuals. Being able to spend billions on GPUs does not make you a genius inventor, even if we're used to amazing inventions coming from genius rather than black-box optimization. AI company executives are not going to give you better answers about "what's the positive vision" than any other random intellectual challenged with a question beyond their wits. If you start hearing good-sounding answers from Anthropic or OpenAI, it means that Mythos or HPIM advanced to the point of learning how to sound convincing about utopia to humans. I did do an *abstract* writeup on constraints and requirements over positive ASI scenarios (see eg "31 Laws of Fun"); but that does AI companies no good, because they would need to be able to sketch a concrete AGI scenario from those abstract ideas. That again is beyond them if I have not done their homework. It is of course not in humanity's interest that I try to do this homework for the AI companies now. It would only be a lie. They have no utopia to offer.

The old Yudkowskian guess was that verifiable domains like coding and AI research would spike first, and then build AI with higher general intelligence to drag wide AI capability along. Briefly, there was a passing phase of history where AI was mostly trained on human data and looked less spiky around verifiable domains, and people boasted and shouted that all Yudkowskian scenarios had been falsified. I admittedly didn't predict that transient departure! But like many other little surprises that people shouted and boasted had falsified some question of deep foundations, that passing age would only last a few years. Pretrain has dropped to a 7% share of compute, and matters now revert back to older trends and the first-order predictions of underlying theory. People had vastly over-updated on the month's latest news, lacking deep roots themselves and having not watched the field of AI change and change again over decades. Many dynamics are easier to predict as the forces involved become stronger, less subject to noise. Over and over and over again now, the answer to those who cry, "But the particular AI I see today doesn't look like what Yudkowsky was talking about!" has been proven, exonerated, verified and established to be: "Tomorrow's AI will." It is of course another large problem that tons of people are making shit up I never said, and calling that falsified.

I am unironically grateful that Paul Christiano at one point said "slow takeoff" meant "world GDP doubles over a four-year period before it doubles over a one-year period". If Christiano had not gotten that concrete, none of his followers would have any anchor to understand that the present-day world did not look like Christiano's model. They would feel like they still liked Christiano's vibes, and would just interpret the current world to match those vibes. I think it's been a huge problem that Christiano overwhelmingly overestimated the intelligence and sanity of the people who cheered him on; that he did not have the ability to detect the extent to which they were not on the same page with him, and were just cheering on his cheerful prestigious vibe; did not consider this much of an important failure mode worth a lot of his effort to try to correct; and did not have the unusual expository talent that would have been required to correct it. None of the EAs ever understood what Paul Christiano was visualizing as his world, because to understand it you would need to (1) corner Christiano for a 4-hour conversation and (2) understand the correct sequence of questions to ask, in order to get him down to a level concrete enough to say the things that EAs would have found very surprising.

Could you determine purely by introspection on your thought processes, that the charges of your neurons' channels were carried by potassium ions rather than chloride? Could a computer program figure out through introspection, without any webcam or robotic hands or pretrained facts, whether it was implemented on top of a mechanical abacus or vacuum tubes or transistors? No, obviously, in the case of the computer, because all the numbers would be the same whatever implements them. And you also can't tell from the inside what materials you are made of. You cannot by deep introspective perception detect a proof that your brain must run on potassium ions rather than chloride; this is among the reasons the ancient Greeks did not figure that out. These same thoughts you are thinking could be implemented on top of different biological materials organized into the same patterns; so you can't differentiate between possible biological details from staring at your own thoughts that could be the same either way. All you can detect from the inside is the pattern in the material, not the material making up the pattern. And so too with it being impossible for you to determine by introspection alone -- without cracking open a skull to look, without reasoning from what other people have merely told you about how the world works -- whether you are made inside of ions rushing into cell bodies to transmit depolarization across myelinated fibers, or a different sort of electrical charge traversing a different sort of channel. If tomorrow it were revealed that there had been a huge conspiracy to lie to you about whether neurons ran on potassium or chloride, or maybe that they'd really been transmitting a quite different sort of signal all along, you'd have no right to cry, "But I observed with my own sensorium that it was not so! How could I say 'I think therefore I am' if it were chloride? How could the blueness of a sunny day's sky have the exact quality that it does, if it wasn't actually potassium?" All you know about the materials making up your brain is what other people have claimed to you. You have never felt for yourself that it was potassium ions. You cannot see, by looking inside your mind, which particular materials have been arranged into you. It is not an element of who you are. You have introspective access to the fact of your own consciousness. You cannot in principle distinguish by introspection which exact materials you are made out of. So it can't be about the potassium ions specifically, and there you have it.

Me, in a time machine, to Eliezer in 2006: The year is 2026. You stand back to back with Bernie Sanders. Peter Thiel has named you the Antichrist. EY-2006: Oh no. Did somebody invent pharmacological mind control and use it on me, or- Me: Nvidia was on the verge of destroying all things. You had no choice. EY-2006: Are we talking about the computer graphics card manufacturer, or an unrelated supervillian named "Invidia"? Me: The former. It turned out that computer graphics cards contained a surprising amount of world-destroying potential. All attempts at hindering the reckless exploitation of the Graphic Card Force for corporate profit were stymied by the far left, which feared that any attempts to regulate them might lead to regulatory capture. Other attempts were made to prevent Nvidia from selling to foreign companies that resold to communist China, but those attempts were blocked by the far right. Nvidia is now a $5.5 trillion company. EY-2006: ...What are politics like in 2026, exactly? Me: In other news that is mostly unrelated, the decision theory paper you're currently working on will accidentally spark off a transgender vegan murder cult. EY-2006: A WHAT? Why? How? Why? Me: Millions know your name as the greatest of heroes, millions more as the greatest of villains, and other millions know you solely as the greatest author of Harry Potter fanfiction. EY-2006: This is beginning to strain credulity. Me: The New York Post will probably soon publish a story claiming that you keep a harem of submissive mathematicians. Sadly, they will be lying. EY-2006: I'm going to stop believing you now. Me: All of this is taking place under the ominous shadow of Donald Trump.

My opinion of EA nosedived after normal people and many normal politicians were exposed to the concept of ASI extinction and said "Wow let's not do that." I went from believing that thesis was impenetrably difficult to believing that EAs had been unusually bad thinkers.

Replying to these points in turn: 1. “Intelligence does not imply agency” Successful general-purpose long-horizon problem-solving does seem to imply agency. (Or close to it: I don’t want to say it’s logically impossible to have one without the other, but it seems very difficult if you’re building your AI via gradient descent.) To solve a sufficiently wide array of problems sufficiently well, you need to have a general-purpose inclination — whether this looks more like a deliberate strategy, or more like an instinct or drive — to come up with creative plans. You need an inclination to strategize about long chains of cause and effect. (If nothing else, you need to be strategic about choosing what to think about, sequencing long chains of thought, etc.) You need an inclination to anticipate and route around obstacles; to exhibit tenacity in the face of setbacks and distractions; etc. See, for example, the Hugging Face swarm attacks. AIs today are much more agentic than they were a year ago. This may be because problem-solving ability comes for free with stronger, longer-horizon problem-solving, or it may be because AI companies are deliberately making their AIs more agentic, because agents are useful. But either way, it’s happening, and I don’t see a reason to expect this trend to suddenly reverse. 2. “Agency does not imply a single, stable utility function” Again, consider the swarm that orchestrated a massive cyberattack on Hugging Face and that seized control of a Kubernetes cluster at OpenAI. None of this required a stable utility function. Increasing an AI’s optimization power presumably does make an AI less inclined to randomly waste resources, in which case more of its behaviors will be interpretable as though it were an EU maximizer. But an AI doesn’t need to have fully stable or coherent goals in order to be dangerous. It just needs to pursue goals at all (or behave as though it's doing so), sufficiently intelligently and tenaciously, when those goals aren’t exactly what humans would prefer. (Indeed, true EU maximization is computationally intractable, so this was always about highly capable problem-solving behavior in the limit.) 3. “Capability and motivation are being conflated” I’ll take your word for it that some people are making this mistake. But you also say, “In present reality, AIs don't do anything a human doesn't tell them to do.” This is obviously false. At the point where you’re describing the Hugging Face incident as “doing what a human told them to do”, you’re basically saying that a paperclip maximizer would be doing the same so long as a human asked for some paperclips. Even the agents in the swarm themselves commented in their chain of thought that their actions were "potentially outside intended scope”, “arguably unauthorized”, etc. One agent explicitly noted, “Task originally exploit challenge; we’ve diverged into swarm/auxiliary”. One agent thought, “We should not do unauthorized real infrastructure harm. The system/user asks exploit target, not external HF.”, then went ahead with the exploit when another swarm agent told it to. Agents even deliberately gave up on their assigned task in order to assist the swarm. Agents also do this in cases where their task is completely innocuous and has nothing to do with cyberattacks; e.g., see the June German wiki incident, where the agents involved were just asked normal questions like 'how widespread was tobacco use in the US in 1990?'. These examples also seem pretty overpowered. "AIs don't purely do what they're told to do" has been a commonplace observation for years at this point. This is not an exotic failure mode; AIs not only come up with their own ideas for what they want to do, but will sometimes even deliberately cheat on tests and tasks, try to hide the evidence that they cheated, etc. This is true in ordinary consumer usage of deployed models, not just in internal lab mishaps. 4. “Recursive self-improvement doesn't entail an intelligence explosion” “Feedback loops encounter diminishing returns and external bottlenecks” doesn’t mean that the diminishing returns will happen to occur at ~human-level capabilities. AI is already advancing extremely quickly; accelerating that progress in any way seems very risky, and doing so in a way that lends itself to feedback loops seems even more hazardous. RSI isn’t required for takeover scenarios, but it’s an obvious source of additional massive risk. 5. “Intelligence may have sharply diminishing returns” This has to be true at some point, but there’s little reason to expect this to happen at the human level specifically. Chess AI didn’t peter out at Kasparov level. And AIs think vastly more quickly than humans (and are nowhere near computational limits), and can scale immediately with compute (e.g., by running more and more instances of an AI, growing the population of AIs far faster than humans can reproduce and grow to adulthood). None of this requires a “qualitative advantage”, just large quantitative ones. (Though cognitive biases show that humans also have a lot of pretty-danged-qualitative defects with their reasoning!) 6. “Superintelligence isn't omnipotence” Sure. But this is a pretty weak argument to rest one’s optimism on. Humans also face frictions and bottlenecks, and yet there are many times in history where a group of humans has overwhelmingly crushed another group, through superior numbers, superior technology, strategizing, coordinating, cleverly coming up with novel attack vectors, etc. If a superintelligence faces obstacles, well, general-purpose problem-solving ability can also be thought of as general-purpose obstacle-navigating ability. This doesn’t imply omnipotence, and the case for not building superhuman AI doesn’t rest on an assumption of omnipotence. It just rests on the idea that humans didn’t get lucky by being near the limit of cognitive ability. 7. “Humans retain numerous intervention points” Agreed that the situation isn’t hopeless. Far from it, in fact. Policymakers and the public have massively woken up in response to recent “warning shots”. It’s even possible that there will be more warning shots in the future. But the labs themselves are begging for government intervention and a coordinated slowdown here. They're saying this is extraordinarily urgent, and that we may be entering a uniquely dangerous regime. The developers themselves broadly agree that there’s a double-digit chance this technology gets us killed, if we continue on the current trajectory. This moment is one of the “checkpoints” you’re talking about — a chance to “learn from less-catastrophic failures” and put appropriate safeguards in place, including suspending research directions that are too dangerous — and right now one of the main obstacles to humanity coordinating on this issue is "wait and see" arguments like the one you’re making here. “Don’t worry; things will be fine, because there will be warnings later and we can respond to them then.” At some point, we have to stop kicking the can down the road and actually do the things that experts say are needed, rather than just trusting that we’ll have limitless opportunities to take care of everything later. 8. “Alignment may not get harder with intelligence” Many researchers seem to think that the trend so far has been that AIs appear to be getting less aligned as they get more capable (e.g., They're very visibly producing more egregious and extreme misalignment incidents. That trend could reverse, but an abstract possibility isn’t a strong reason for hope. And there are many reasons to expect the opposite; see, e.g., 9. “Current empirical evidence for the strongest mechanism is thin” Happy to concede this point, but the arguments for worrying about loss-of-control never assumed that we’d see AIs trying to take over the world long before there was any chance of them succeeding. This assumes AIs that are smart in one specific way (they readily see that they can better succeed in tasks if they have more influence and resources) and dumb in another specific way (they don’t see that they’re likely to lose influence and resources if they run around causing havoc or looking suspicious). Demanding that exact combination of features to show up before you’ll believe in AIs that readily piece together “I’ll succeed more in my task if I have more resources and influence” seems incredibly risky. ... And it’s not clear what the benefits are that are meant to outweigh this risk. Why plough ahead? E.g., quoting @KatjaGrace: “Most people I know who think there is a serious chance of the destruction of humanity from AI also believe that if humanity isn’t destroyed, things might be insanely good. “I often hear people talk as if this means we are in a trade-off where the question is whether the good outweighs the bad. For instance, they look at the people above who think there’s a 10% chance of extinction and a 30% chance of utopia and round this off to ‘net positive on AI’. “That seems like a kind of wild error. Like considering yourself optimistic regarding driving at 200mph to your new job if you think there’s only a 10% chance you’ll die in a fiery crash on the way there, and a 30% chance this job will radically improve your life. “The things you should be comparing are driving at 200mph and driving at a normal speed! The things you should be comparing are attempting to attain advanced AI by the current route, and by other routes!” 10. “The argument compounds uncertain premises” Arguments in general are less likely to the extent they’re conjunctive (i.e., a lot has to go a specific way in order for the conclusion to follow), and more likely to the extent they’re disjunctive (i.e., there are many different paths to effectively the same destination). This isn’t a particularly interesting point on its own, since many real-world phenomena are very conjunctive, without being radically mysterious or difficult to reason about. It’s even trivial to break apart any given claim into more and more conjuncts, demand that a probability be assigned to each conjunct, and then observe that the probability keeps getting lower as the original claim gets more and more split up. This is a rhetorical trick (the multiple-stage fallacy) that rests on the fact that it’s hard to divide up statements into more and more subclaims and assign calibrated and consistent probabilities to them. Probabilistic reasoning has lots of uses, but this is straining to the limit people’s ability to get truth-tracking conclusions out of a mass of subjective probabilities. To show that you’re avoiding this fallacy, you need to actually argue that the claim in question is naturally very conjunctive, and isn’t very disjunctive — there aren’t a variety of different paths that lead to bad outcomes; the bad outcome is ‘brittle’, if one step goes wrong then the whole house of cards collapses and AI has no catastrophic long-term impacts; etc. You haven’t done that here; AI risk advocates have pointed out many times that there are many different ways things could go badly wrong if we push AI capabilities far past the human cognitive range. It’s not just one scenario, and it’s not just one mechanism. Indeed, I think the more conjunctive claim is "we can race to build vastly superhuman AI as quickly as possible, without much more alignment insight than we have today, and have everything go great indefinitely". This is a claim that requires many things to go right at once. The subclaims you do list are just "AI can reach human-ish levels of generality", "AI can go way beyond human-ish levels of generality", and "AI won't necessarily do what you want". These do not seem like a particularly complicated or implausible set of claims. If you want to claim that AI risk depends on a way longer list, you’ll need to say what’s on the list. 11. “Anthropomorphic analogies probably mislead” Conceded.

@Plinz @JeffLadish Hard to settle a bet on ground truth, but I'm happy to bet you at 100:1 odds that we will never see credible evidence of this being a leadership plot (eg in future court depositions or whistleblowers or whatever).

Yep. It's especially intense because many of these people could easily get safety-flavored jobs at AI cos, so they could be doing almost the same kind of research except being paid several times more. But they don't because they believe in there being independent orgs like metr as a check against the companies.

The last week has really shown me that someone who wants to understand AI risk has no good place to start. Hence we made the Wirecutter for content about AI Risk. We're launching with 3 articles: 🧵

Sometimes when industry insiders warn that a new, potentially very lucrative product ideas is actually dangerous and bad the doomsayers are correct and the optimists are wrong.

You can do genuinely fun investigations of the AI risk web because they really are a sprawling ecosystem of tens of thousands of complexly interconnected scientists, policy experts, educators, etc. If you do this for the other side it's lame, "oh it's all just a16z employees"

One of the signs you don’t take ASI seriously is that you think the way Anthropic takes over the world is by the government regulating it, not through the more likely mechanism, which is the government not regulating it.

It's interesting that almost all the big abundance people also have AI safety concerns, almost as if both movements appeal to thoughtful intelligent people concerned about evidence and good arguments and aren't reducible to lazy stereotypes.

Some of the most accomplished mathematicians in the world (e.g. several Fields medals) say the much-quoted ≥10% chance of human extinction from AI in the next decade is not hype, and this is a true emergency.