This is why I think “we’ll just get ASI and tell the AI to solve climate change for us” is the most likely way a Real Paperclip Maximizer scenario manifests: the AI concludes that human civilization is the main cause of climate change and deletes it.

Maybe you have recently become aware of the AI safety debate and the arguments swirling around it. If you want to understand them, you need to understand a couple things that almost everyone gets wrong: There are TWO distinct classes of AI dangers, and it's VERY important to think about them separately, and not let one confuse you about the other. Many many people (including many quite intelligent, clear-thinking people) do not effectively understand the fundamental differences between these two classes of dangers. The first one has to do with the theory that an AI far superior to human intelligence (Artificial SuperIntelligence, or ASI) will inevitably wipe out the human race. The second one has to do with the idea that powerful AI will result in very harmful things happening to many human beings, possibly all human beings. Those two sound VERY similar, don't they? They are DIFFERENT. Understanding how they are different is crucial if you want to think about or contribute usefully to any conversation about AI safety or AI harm. You might feel like you are Making Very Good Points or Asking Incisive Questions, but if you aren't clear on the differences between the two, you aren't. So, I'm going to tell you what the difference is so that you can talk more usefully. The first one concerns itself with a very specific thing, which is ASI (Artificial Superintelligence) that is more intelligent than any human being. When I say that, I am not referring to a thing like how Einstein is smarter than you, we are talking more about something like how a human being is more intelligent than any mouse. In our regular lives, we meet other people who we can tell are smarter than us, vs some who are less smart. The line is fuzzy, because intelligence has a lot of dimensions. I'm better at a "rotating shapes" kind of intelligence than my wife, and she is better at "words-making" kind of intelligence than I am. But every human is better in almost every dimension of intelligence than every single mouse. That's the level we're talking about: an artificial superintelligence - made up of a computer or a network of computers - that is more intelligent than any human. And more intelligent by a long shot, by a wide margin, in an indisputable way like how humans are above mice. That is the first thing. The theory says that if you have an AI that is vastly smarter than all humans - in the way that a human is smarter than mice - that superintelligent AI will inevitably, eventually, sooner or later, wipe out every human on the planet. We will refer to this as "existential risk." The common follow-up question "well, how exactly is it going to do that?" is NOT the important question, and one of the most important elements of understanding this theory is first getting why that particular question is not important. A couple analogies: Analogy 1: You are playing chess against a grandmaster. My theory predicts the grandmaster is going to beat you. You can ask "Well, how exactly is he going to do that?" I don't know, because I'm not a grandmaster, I just know that a chess grandmaster is almost always going to beat a normal player like you. And I'd be right. So the question "how is he going to do that" is not important, and doesn't affect the final outcome. He's going to figure out a way because he's way better than you. Analogy 2: Humans are smarter than all other animals, comprehensively, by a wide margin. We have driven numerous species to extinction, not because we hated them or hunted them. Many of them have died out without most humans even ever thinking about them. All we did was expand our civilization, use up resources, encroach on habitats, and pretty soon the resources needed by those species went away and they died out. We figured out a way to get what we wanted because we're way smarter than them, and often we didn't even notice they died as a result. A lesser animal asking, "how are the humans going to wipe us out?" is not asking a relevant question. We don't know, but we do know that any time humans and lesser species compete for any kind of resources, the humans will win. The fact that we know who is going to win beforehand - and that it is due to the vastly different levels of intelligence - is the key concept here. A vastly more intelligent AI is likely to care about things that are incomprehensible to us, the way animals can't understand human goals. It's going to need resources to pursue those goals and it's going to be far more effective at gaining control of them and excluding us from them - in the same way that we are far more effective than other lower species. A much more intelligent AI will not care about our interests, it will care about its interests, and to whatever small degree we happen to escape total annihilation from losing access to all our resources, any remaining humans will likely be enslaved into a system that serves the AI's own purposes. That is the first thing. (Remember how I said at the beginning of this post that there was a first thing, and then a second thing?) The first thing is the most difficult to understand, because you have to extrapolate how a vastly superior intelligence would act, and you can only use analogies like "how do humans treat lesser creatures," and the analogies are messy. But now let's move on to the second thing. The second thing is "everything else you've ever heard that AI might do that's harmful." That's a little inaccurate. It's actually "everything else you've ever heard that humans might use AI to do that's harmful." This is the critical difference. The first one talks about the inevitable outcome of what happens when two vastly different levels of intelligence collide, e.g. ASI vs humans, or human vs mice. The second one has to do with what happens when humans possess AI as a powerful tool. This second thing is much easier to understand, because we have many more concrete notions: Like: - the military uses AI to make hyper-efficient killer drones and missiles - your capitalist overlords use AI to replace you and everyone loses their jobs - authoritarian government uses AI to surveil everybody and control the entire population - hackers use AI to break into secure networks and hold companies and governments hostage - students use AI to cheat on homework and show up to college knowing nothing - AI slop saturates the internet and makes it impossible for artists and writers to make a living - terrorists use AI to make biological or nuclear weapons or even things like - the military hands control to an AI and it misinterprets something and launches nuclear attacks and kills millions All of those sound pretty familiar, right? Yeah, you've heard them before. We call this second thing "risks from misuse." These problems are not the first class of problem! This second class of problems exists while AI is a tool that can be controlled by humans, and humans use it to do evil or careless things to each other. The problems may sound exotic or dystopian or novel, but they are fundamentally problems having to do with flawed human nature. Given a powerful tool, some humans will likely use it to control or otherwise harm others. This is a very familiar problem. I am not condemning or condoning this. I'm just describing it. That is a fundamentally different danger from the first thing, which is that when a human is far superior to a mouse, the mouse is likely to come to harm because the human cares about doing human things, and the mouse is not gonna make it once the humans get going. ===== Hopefully from the above, you have understood the difference between the first thing and the second thing. I will list them again - see if you now understand how they are different: The first one has to do with the idea that an AI superior to human intelligence (Artificial SuperIntelligence, or ASI) will inevitably wipe out the human race. The second one has to do with the idea that powerful AI will result in very harmful things happening to many human beings, possibly all human beings. Can you tell how they are different now? If not, re-read the stuff from earlier until you understand. We call the first one "existential risk" and we call the second one "risks from misuse." Once you understand, here is the CRUX of the problem: SOLUTIONS TO THE SECOND THING DO NOT HAVE ANYTHING TO DO WITH SOLUTIONS TO THE FIRST THING. In fact, it's worse: Solutions to the second thing (misuse) look roughly like "give powerful AI to as many people as you can, so they can fight the other people using powerful AI." But the general solution to the first one (existential risk) is basically "don't let anyone have powerful AI, no one can control super-intelligent AI." Throughout history, harms from technological misuse typically arise because a small group has control of it and can use it to dominate or harm others. Once everyone has it, things tend to stabilize: you can hurt me, I can hurt you, maybe we test each other (ouch 💥), and then we agree not to hurt each other. But the first one (existential risk) pretty much just arises if anyone (good or bad!) creates a superintelligence. Because they aren't going to be able to control it, the superintelligence will decide it has other priorities, and then we will be at great risk of being wiped out. And the solutions that generally work to solve problems like the second thing are EXACTLY THE OPPOSITE of the ones likely to solve the first thing. THIS is why lots of arguments about "AI risk" or "AI safety" go nowhere. Because someone will be thinking about the risk from the first thing, and another person will be thinking about the risk from the second thing. Both are plausible risks but fundamentally they arise from different things - and so the solutions are not just "bad" or "flawed" - they are likely to be very nearly exact opposites.

Finally our AI leaders (Dario and @Sama) are doing the sane thing. "Pacing the frontier" is exactly the right thing to do: keep moving forward, coordinate the pace, and collaborate on alignment research. It is the right balance. To the people whose objection goes to "but this hands the lead to China!" - no, YOUR ENTIRE WORLDVIEW IS GROSSLY WRONG - and moreover your ability to correctly perceive the world and predict events will always go awry until you correct it. Here is the core of your gross error: - You do not think people in China are actually people. - You do not understand that Chinese civilization is a peer, if not superior, to your own. If you actually thought of Chinese people as real people, with equally valid and real desires, dreams, and actual human existence equivalent to your own, you would realize that: - China is not the Soviet Union, and you'd stop mapping notions of Stalinist authoritarianism onto it - China's current government is valid and broadly supported by its people because it has improved the quality of their lives and continues to be focused on doing so - Western values are not universal, as values are culturally influenced and ultimately validated by their consequences; the West has not been doing so well lately on that front, and so - Your deontological bleating of "but we are free and they are not" is less than worthless: Chinese people most value being free from the actual entity most responsible for their oppression, namely WESTERN COUNTRIES And so, if you can start to break out of the many layers of your brainwashing, you may further realize that: If you are a country engaged in a geopolitical race to develop a key technology that: 1) may strongly determine your country's own future sovereignty, 2) one which almost every country agrees that if they don't have it, they will be subject to domination by other countries who do, and 3) your rival is clearly ahead, and 4) people are starting to realize that unrestrained development may lead to significant global danger, then it should be perfectly logical that your answer to any call to "mutually agree to stop" must start with the player who is ahead agreeing first to slow down. If you can realize that both countries are actual real civilizations with real human beings who have actual real lives who have exercised self-determination when it comes to how they are governed, and not One-Dimensional Cardboard Others, this is perfectly logical. Imagine a scenario where the US and Soviet Union were engaged in a race to build ever more powerful and numerous nuclear weapons, and the Soviet Union was clearly ahead. Then the Soviet Union points out that this risks worldwide nuclear annihilation, and proposes that both countries agree to stop building nuclear weapons and engage in mutual disarmament. The obvious and logical reaction of the US would be to say, "Yeah, we agree there is a danger, but we'll agree to begin disarming once you've disarmed your own weapons until we're at parity." Or, you'd try to accelerate your own development until the US reached parity, at which point you'd return to the table and have discussions about mutual disarmament. Right? You definitely wouldn't agree to stop and disarm when the other guys were AHEAD. Your main danger is that the other guy is ahead, not something bad that lies ahead of them. The argument in the US is "if we slow down, China will get ahead!" Guess what, the real people in China see it as, "the US is already ahead! They're asking us to stop?" The stronger party needs to make the gesture of good faith, if they are genuine. There is real-life precedent to this! Back in the day, when the US proposed to China that we engage in mutual nuclear disarmament, China's answer was basically "We will be happy to engage in mutual disarmament once the US first disarms to the same level of warheads that we have!" For reference: China has ~600 nuclear warheads; the US has ~3,700. If you are a country that was previously invaded and colonized by 8 countries at once, is surrounded by military bases held by your rival, whose neighbor committed horrific war crimes against your people and remains unrepentant and whose pacifism appears to only be maintained by that same rival occupying it... if you were a real human with a real life and real hopes and dreams, you would reasonably expect a superior rival proposing mutual disarmament to be the one to volunteer to stop first, if not disarm down to your level. China currently has inferior models, lacks leading-edge chip technology, resorts to distilling US models to train its models. We have yet to see even a single Chinese model that does anything uniquely Chinese besides refusing to answer questions about Tiananmen. [1] Recognizing that the people "on the other side" are also people means recognizing that they might also care about the same things you do and that they care about their lives and families and civilization and Not Getting Into A War Where Loved Ones Die and aren't going to do the "well a cartoon villain would do the evil thing so that must be what they would do" thing. That is why your entire worldview is wrong. You think you are a real person and you don't think the other people are real people. You think your country is filled with real people, and the other country that's much older and has four times as many people isn't filled with real people. So obviously you are not going to be able to form an accurate worldview. Otherwise quite intelligent and thoughtful people in the US seem to have this blindness (it seems to be bipartisan), and don't really regard China as being a real civilization populated by real people who chose their government and want it to do what it's doing because it's actually good for them. Here's a common "anti-safety-ist" argument you might've heard: "The doomers are fearmongering and trying to make everyone scared of AI, just so they can justify a solution where they control everything and keep us from having abundance!" Sound familiar? Do you believe it? Then maybe you also believe the true version: "Our leaders are fearmongering and trying to make everyone scared of China, just so they can justify a solution where they control everything and keep us from having abundance!" If your reaction is "but no, China actually is... " maybe you want to realize how much you've bought into what you've been told. Maybe you're not aware of how many high-quality Chinese products you could affordably have access to. Do you know what abundance requires? Overcapacity. The people telling you that are the same leaders who have been shown to lie to you about . Why do you somehow think the one thing they're telling you the truth about is that China is a threat? It's only a lie they can maintain because you readily believe that Chinese people in China aren't real people, because if you thought they were real people, fellow human beings, you'd realize that those lies could not be true. Chinese people are the most unruly, ungovernable people in the world. Chinese history is basically a history of overthrowing their governments - if they have a government that is popular it is because it is doing quite well for them - though it may seem weird to outsiders because Chinese people are pretty weird. I know this, of course, because I'm ethnically Chinese and I know actual Chinese people, and at the same time I was raised and educated in the US so I know actual Americans, and everyone is Real People, not cartoon Cold War villains (who, when the Cold War ended, we learned were also real people with hopes and fears and dreams). There is, overall, strong willingness to work together there - cynically, because working together is how you get rich, and Chinese people just want everyone to get rich; war does not make everyone rich. But the US should stop being naive and realize that when it comes to AI it has to take the first step in extending the bridge. This announcement and the similar one from OpenAI a couple weeks ago are the right start. ==== [1] Seriously, where's the 5000 years of written training material? Why isn't Qwen quoting Confucian analects or traditional idioms to me when I ask it for advice? Why is it always neoliberal pablum? The US is more dominant in AI than it thinks.

TIL that the annoying tell of AI text is called a "cataphoric teaser" - the linguistic construction used to create a cheap feeling of suspense. You know: "Here's the part that nobody tells you..." "Here's what most people get wrong..." "The part most people sleep on..." A decade ago, we knew this as: "You don't believe what happened next!" "Doctors hate this one weird trick!" Alas, we will never escape the dystopia of our own making.

Wow, TIL that the reason the chess queen is powerful was because when chess reached Spain in the 15th century, Queen Isabella I changed the rules on how the queen moved (previously it was highly restricted) and the new version was called "Mad Queen Chess" - which proved to be so popular (it made the game much more dynamic and exciting) that it became the dominant ruleset.

A key stat that every environmentalist who is serious about solving climate change should know: The US Navy has put into operation over 400 nuclear reactors without a single failure that resulted in any radioactive release. That is equivalent to the entire total number of civilian reactors deployed worldwide, but the Navy’s reactors are literally deployed in BOATS IN THE WATER, that continually move, rock, get shot at, etc. There are non-trivial engineering challenges, but the core reality is that safe, clean, abundant nuclear is a matter of political will, intelligence, and organizational culture.