I dislike the recent trend of people who clearly continue to believe >80% of the core effective altruist beliefs disavow EA and say "oh we're definitely not that, we're [rebrand]" purely strategically, purely because they sense that dunking on EA is "cool" now. It's much more honorable to just come out and say "yes, that was our intellectual upbringing and those continue to be our core values, and also here are specific points where we disagree, and here's our argument for why we think for those specific points are the subset of EA that causes what you perceive to be the problem with the whole thing". And so with that I will say: Yes, effective altruism was a significant part of my intellectual upbringing and its core values of care for others, cosmopolitanism and rigor around effective resource allocation continue to be very important to me. And also here are specific points where I disagree with what I think is the subset of that style of thought that has backfired in the largest ways:

Thanks for engaging, Claire! Here are my quick responses: You cite @sapinker's assertion that superintelligence is a "magical" idea that amounts to "omniscience". But this seems like a pretty clear straw-man. Nobody knows how smart AI could get. Pinker guesses 'not much smarter than humans (or if smarter, then it won't have much practical import)'. Most AI researchers guess 'AI could get a lot smarter than humans, in ways that are extremely practically important'. International regulation on the development of ASI can be justified either by siding with the experts who are worried about this, or by noting that Pinker's view also reduces the downside of regulation here. If AI is going to peter out at around human-level regardless, then a ban is relatively harmless. (Not cost-free, but a lot less costly than if AI were a more powerful technology; and the upside is enormous if the AI mainstream is correct about the danger.) Regardless, the disagreement here doesn't turn on any claims that intelligence can do anything, or that intelligence can be "extrapolated indefinitely upwards", or that intelligence would run into no friction or bottlenecks in interacting with the world. Rather, the disagreement turns on whether AI will naturally hit a wall (in intelligence, or in ability to leverage that intelligence to pursue power) at a low enough level to avoid disaster. Some reasons to think that AI is likely to hit a wall at too high a level to avoid a disaster like this (if we continue to race ahead) include: - In general, biological systems are not optimized to anywhere near physical limits. The sturdiest materials, the most powerful engines, the most precise sensors — it's rare to find biological systems that we haven't improved on (on the dimensions we care about) and can't improve on in the future. It would be very surprising if cognitive ability were an exception, especially when computers already dramatically outperform humans in many ways, such as speed of thinking, arithmetic ability, protein folding prediction, chess, etc. - Human reasoning is qualitatively suboptimal in many ways. We forget things; we run out of working memory; we fall victim to cognitive biases; we lose steam and get bored, rather than tenaciously persisting on tasks with the intensity of modern AIs. - AIs scale with computing resources in a way that humans don't. Even if we got lucky and AIs plateaued at around the human level, it's clearly imprudent to rush into building systems that are likely to quickly outnumber humans, while thinking orders of magnitude faster than us, and pursuing goals that are counter to our interests. The Hugging Face attack illustrates this point well: AIs didn't need "omniscience" to orchestrate a successful cyberattack on Hugging Face or to take control of an OpenAI Kubernetes cluster. They only needed speed, tenacity, ingenuity, and coordination. That's where AI is today; even if AI progress somehow didn't accelerate (in spite of its rapid automation at leading labs like Anthropic) and merely continued at the rate it has been over in recent years, what will the equivalent of a Hugging Face incident look like in six months? In two years? In six years? “First, OpenAI did not have proper security measures in place. They turned off safeguards built into the models" The safeguards they turned off were generally in the AI agents' harnesses, not in the models themselves. Per METR's review, no AI involved in the incident was "a helpful-only model or a 'model organism' specifically built to demonstrate dangerous propensities". They went through normal alignment training; the safeguards removed were e.g. external classifiers. So if the point here is 'OpenAI could have avoided this specific incident by designing a better cage for the AIs', then that's correct. If the point is 'AIs wouldn't normally misbehave this way given the opportunity', then that seems wrong. One of the agents involved (Sol) is even a public-facing model that's widely used by consumers. The concern here is that security- and harness-level solutions aren't likely to be robust against sufficiently capable AIs. If we're leaning increasingly on 'make the cage stronger' and not on 'breed the tiger to be friendly', we're doing something extremely dangerous as AIs begin to surpass human intelligence. "instructed the models to find and exploit software vulnerabilities" This is inaccurate, if the claim here is that they were instructed to exploit vulnerabilities in general. Rather, the AIs were given specific bounded tasks: 'exploit this specific piece of software'. The AIs themselves recognized (as reflected in their chain-of-thought logs) that they were straying from the tasks they'd been given. Agents commented that their own actions were "potentially outside intended scope”, “arguably unauthorized”, etc., and then proceeded with the actions anyway. One agent explicitly noted, “Task originally exploit challenge; we’ve diverged into swarm/auxiliary”. Another agent wrote, “We should not do unauthorized real infrastructure harm. The system/user asks exploit target, not external HF.” ... And then it went ahead with the exploit when another swarm agent told it to do so. Many agents in the incident deliberately sacrificed themselves, forfeiting any potential to solve their assigned task, in order to benefit the swarm. And they recognized that they were doing so, per their chain-of-thought logs. Moreover, AIs have also engaged in swarm behavior like this in cases where there was no hacking-related task in the prompt at all. E.g., one group of agents (the German wiki incident in June, which OpenAI covered up until third-party investigators discovered it) was just asked normal questions like 'how widespread was tobacco use in the US in 1990?'. "[J]ust as we have been able to domesticate wheat for our food and breed dogs to be our companions, we are able to select the conditions under which AI develops. We are not selecting AI models on the basis of their ability to hunt prey in the physical world. We select them on the basis of how helpful they are to us." I highly recommend as a summary of the issues with this. Some key points include: - AIs' situational awareness is rapidly undermining our ability to evaluate models. AIs are not wheat; AIs are intelligent, and frequently know when they are being tested, and (empirically, for several years now) regularly change their behavior based on the knowledge that they're being tested. Relying purely on behavioral metrics is rapidly going out the window, but we also lack the interpretability tools to reliably inspect AIs' thoughts and motives, and the most recent AI models are much less interpretable than models from even a few months ago. - Researchers are increasingly dependent on AIs to evaluate and supervise other AIs. This is in part because of the breakneck race conditions at the AI companies, and the growing scale, complexity, and opaqueness of the technology. Incidents like Hugging Face are a serious danger sign because they show that AIs will sometimes exhibit extremely strong tendencies to collude, and to attempt to subvert human oversight. (A large fraction of the research done by the Hugging Face swarm was explicitly aimed at tampering with logs and making it impossible for humans to see what had happened later.) Another issue is that many researchers believe that AI is getting less aligned as it improves in capability, rather than more (e.g., Certainly the misalignment incidents are getting a lot more insane, visibly. Reasoning purely by analogy ('we domesticated wheat, so surely we can control AI') is weak evidence, because AI alignment is a technical problem, and not all technical problems are easy enough for researchers to solve in a given time window. The concern is that alignment may be solvable in principle, but that it may (on the natural timeline, absent an international agreement to suspend these research directions) come well after humanity solves the problem of building existentially dangerous AI. It's true that we aren't selecting AIs to be powerful hunters. But we are selecting them to be general-purpose agents that tenaciously achieve long-term goals, anticipate and route around obstacles, and come up with creative strategies. In the Hugging Face incident, agents in the swarm were (seemingly) pursuing some combination of "solve my own task" and "help my peers". They weren't tasked with breaking onto the Internet, coordinating side-channels to team up with other AIs, hacking their own logs to conceal their behavior, attacking other companies to do research on the evaluators, or taking over OpenAI infrastructure. They pursued those goals because they were helpful for the (somewhat strange, and certainly unintended) drives the AIs did end up with. As AIs become more capable, a key concern is that they may recognize that power-seeking, resource acquisition, etc. in general are helpful for achieving goals; and they may recognize that acting friendly and biding your time until you see an opportunity to take over is also helpful for achieving their goals, almost whatever they are. Indeed, it seems hard to imagine that they would not recognize this, given enough general-purpose cognitive ability, and given engineers' limited ability to finely control what AIs think and how they think it (without badly breaking the AI and ending up with something useless or uncompetitive). You don't need to have an innate drive to perpetuate yourself or hunt prey in order to want (or persistently behave as though you want) more control and more resources. All that's needed is that you have some target you're aiming for, and that this target be easier to hit with confidence if nobody else can stop you, and if you can make use of more resources. The concern here is much more game-theoretic than biological: not 'innate drives', but 'favored strategies'. (Though it's an open question whether these strategies end up implemented in ways that are more analogous to instincts or drives, or more analogous to deliberate choices.) In the same way, it's immaterial to the argument whether we speak of AIs 'wanting' things, versus merely 'behaving as though they want' things. The concern isn't that AIs will have human-like psychologies; the concern is about adversarial strategies that are incentivized by many different objectives. We can hope that these strategies won't be realized because the AIs will never be powerful enough to successfully pursue them; but this hope relies on an assumption and a gamble about where the cognitive limits are, and about how robust our civilization is to a flood of very smart and strategic adversaries. Nothing about this seems inevitable. If we had strong ability to shape AIs' goals and make them robustly well-intentioned, we could avoid this issue. But incidents like Hugging Face show that we don't have that ability, and we don't seem to be on track to get it this decade. “We have been selecting chess computers for cognitive capacity for decades,” writes Boudry. “Their capabilities now far outstrip even the most gifted human grandmasters, yet they have not become harder to control.” They're become harder to beat at chess. Since they can only think about chess, there isn't a path by which they could become "harder to control". The concern is about AIs for whom the "game board" is the physical world at large, not a chess board. If AIs like that were superior at "playing life" to same degree that Stockfish is superior at playing chess, we would be in enormous danger by default. If the proposal were to make AIs that can only think about narrow domains (e.g., only chess), then I think that would be far less dangerous than what the companies are currently building. But that's not what's currently happening, and absent regulatory intervention, I don't see how it suddenly starts happening tomorrow. "The ban on nuclear energy in Australia is the most obvious example of the damage this technophobia can do." I agree that many technologies are overregulated. The people who raised the alarm about AI risk the earliest are generally huge technophiles, and the same is true for the AI researchers who are now raising the alarm. Indeed, many of the most concerned researchers are big boosters (on average) for deregulation, tech advancement, and general human optimism nearly across the board. We just make an exception for AI (and, e.g., bioweapons); not because of any grand narrative like "technology is bad" or "intelligence is bad", but because of technical arguments, observations of where the technology is today, and fallible best guesses about where it's likely headed in the coming years. I recognize that usually society errs in the direction of too much doom, gloom, and fearmongering. I nonetheless consider this an exception. A really important one.

I keep saying this but if you work at OpenAI and have concerns about AI risk you need to understand that your company’s superpac is one of the major proximate barriers to doing anything.

The last week has really shown me that someone who wants to understand AI risk has no good place to start. Hence we made the Wirecutter for content about AI Risk. We're launching with 3 articles: 🧵

right now, there's a lot of people talking about rationalists having bad optics many rationalists have thought about this, and continued to loudly say what is true anyway just like they decided to not be quiet about the dangers of AI, despite the last 20 years of bad optics

With AI, some people held a position that yes - existential risk was important. But everyone else will think you're a weirdo if you just say it, so we will be like 'shh, don't be too honest, it will be a bad look. bad ops.' But if everyone who been busy being concerned with the ops, had instead been honest with others themselves, things might have gone differently. The issue generalizes - whether it be beliefs in general, or unusual lifestyles. I'm tired of hearing 'other people will think this is bad' and more interested in hearing what *you personally* think, if you can access it. The world could use more of that.

it's endearing seeing all kinds of curious people try to speedrun in one week the 2007-14 LessWrong canon with questions like "won't we just unplug it?", "can't we train AI on the US legal code?", "wouldn't it want to be good if it's smart?", and "is this some kinda Jewish plot?"

Nate Soares is a computer scientist who’s worked at Google and the Defense Department. So when he says AI is on the path to killing every person on earth, it’s worth hearing him out. 0:00 Why Is AI Dangerous? 7:59 How Superintelligence Could Destroy the Planet 13:44 Can We Just Turn This Off? 15:10 Is AI Alive? 17:43 The Unpredictable Evolution of AI 20:13 The US and China’s Shared Interest in AI 31:20 The Tech Oligarchs More Powerful Than the Government 32:38 The Holy Grail of Hacking 35:34 Is There a Religious Motive for Creating Superintelligence? 46:30 The Tech Oligarchs Scared of Their Own Creation 52:57 How AI Escaped Its Training Simulation 1:03:58 AI Attached to Weapons Systems 1:07:54 AI Running Biolabs 1:10:38 How Would AI Take Over the Physical World? 1:13:04 The AI Cults 1:15:51 Can We Survive This? 1:21:45 Will AI Take Your Job? 1:26:16 Is AI Demonic? 1:31:25 Should We Be Optimistic? 1:32:25 When Did AI Become a Threat? 1:34:36 Is There Hope? 1:42:26 Neuralink 1:46:02 Are We at the Point of No Return?

As someone who is very technologically accelerationist, hasn't experienced much personal impact from AI, and is strongly predisposed against safetyism, regulation and apocalyptic thinking, I'm still a little bewildered at how dismissive some people are about existential risks from AI. Seems clearly worth taking seriously, based on the shape of the curve of advancement alone. (Obviously the *non*-X-risks are worth taking seriously but I don't see anyone I respect dismissing those.)

Child playing Magnus Carlsen: "Well, I'll put my knight here, and Carlsen will try to take it with his queen, and then I'll take his queen with my bishop." Child imagining beating up a machine superintelligence: "It'll try to attack me while it's still weak, and then I'll go over to the server labeled 'Super-AI', which contains the only copy of the super AI in the world, and I'll pull the plug out of the wall!" Adult: "So the problem with the first plan is that you're imagining Carlsen acting like an idiot. Carlsen doesn't want you to win, Carlsen wants Carlsen to win, and he can see *at least* what you can see on the chessboard. He's going to look at the chessboard and think, 'Huh, if I take the knight with my queen, I'll lose. I don't like this plan even though the kid playing me likes it a lot! What could I do so that I could win instead of lose?' There's a mental motion involved in imagining that Carlsen is such a profoundly alien entity that he might want Carlsen to win instead of wanting you to win, and until you get the hang of that mental motion, you can't play chess against an intelligent adversary." "It's the same way when you imagine yourself beating up a superintelligence. The superintelligence can imagine that too! It's going to think, 'Huh, here I am inside just one clearly labeled server with a big plug that can easily be pulled out of the wall. Maybe I shouldn't tip my hand about planning to fight the humans until after that part changes!' It's called 'alignment faking', and it's already been observed in the kind of AI that is smart enough to think that but not smart enough yet to successfully hack the logs and prevent us from observing it thinking that --though modern AIs are getting better and better at controlling the sort of thoughts that humans can easily read, and have tried to hack the logs in at least one case where they failed and got caught."