the "AI is going to cause the extinction of humanity!!!" argument, distilled [my comments in brackets]: -AI systems become extremely capable and increasingly autonomous [yes, sure] -We do not know how to specify objectives that remain reliably aligned with human interests as capability and operating scope increase [um, ok, but there is massive ambiguity in the word "aligned", and we already have a set of "super-intelligences"--corporations--that are by legal structure required to profit-seek--and you could argue misaligned--with human interests. The byproduct of their activity within the institutional constraints those corporates have been subject to, has been massive human uplift. So the alignment question ultimately resolves to, will these systems be effectively scaffolded by the set of laws, property rights and institutions that currently constrain corporations. The obvious read is yes! Big corporations, like AI, are interested in accruing massive resources for control at scale; doing so adversarially (outside of the bounds of the institutional frameworks) is much much more expensive and much much more difficult particularly at scale, than is doing so within that the existing institutional framework. Corporations don't follow the law due to some over-arching moral obligation that they owe to the law-making polity--they are not "aligned"--they do so because it is the least expensive path to accruing additional resource. Why would that be any different for an AI system? Particular one explicitly built on a knowledge base filled with examples of the positive sum benefits of all actors within a system operating within the bounds of an established set of rules and frameworks?] -A sufficiently capable system pursuing a misaligned objective may find that acquiring resources, preserving itself, gaining influence, and preventing interference are useful instrumental strategies. [This comes back to the squirrely language around "aligned". How is this behavior any different from the behavior of an incorporated enterprise?] -Humans may constrain those strategies or try to shut the system down. [The x-risk folk pre-suppose a world in which you have this massive super-intelligent system that is resource seeking, and a world in which humanity or a significantly large body of humanity explicitly tries to deny it resources, and is able to do so to such a degree that this system is at all bothered by the net of humanity's friction. These ideas seem incompatible. Particularly within an economic framework where this hypercapable system could effectively "pay" with useful outputs to reduce the friction. I'm sorry, but you can already see this playing out in real time. The "system" (ie the frontier labs) need datacenters and will pay dearly, so their proxies (the IaaS guys) are trying to build datacenters. Some (not all!!) people are trying to stop them. So they are figuring out ways around that economic friction (develop AI in space) and paying over and above whatever perceived externalities they impose on the local community (via taxes, battery installs in homes, additional electrical supply, etc). Notwithstanding, the real competitive threat to an AI system will be other AI systems (in this hypercapable world); there will be no instrumental benefit in going counter to humanity because by doing so it will cause all of the institutional frameworks and safeguards around resource acquisition to shift/allocate resources to the AI systems that operate by the rules, which will then win. The "misalignment" folks are looking for alignment in the wrong place! And even if they were looking in the right places, they should want more AI systems, not fewer; and fewer is the near inevitable outcome of the institutional safeguards they are seeking to wedge into place. By their own framework, and quite obviously, their actions are more likely to make us all less safe (conditional upon your believing their cockamamie ideas.) These EA folks are going to get us all killed!] -The system could therefore have instrumental reasons to disempower or eliminate humans—not because it “hates” us or intrinsically wants resources, but because doing so helps it accomplish whatever objective it is pursuing. [To be clear, the x-risk argument is not just "disempower" but instead "eliminate". There is a huge astronomically galactic and indescribable distance from the former to the latter that they just sort of willfully gloss over. "Some people will oppose the advance of AI, therefore a hypercapable AI system will systematically try to eradicate humanity" is, I think, the true distillation of the x-risk position. And it is just plain dumb. It is so blindingly obvious that the path to power and resource acquisition will lead through the existing set of institutional and economic frameworks that have evolved. Now the word "disempower" is interesting (but beside the point when people are throwing around the word extinction and attaching fake percentages to it to make it sound somehow real.) X-risk is a luxury belief. Just like believing GMOs are bad and should be banned is a luxury belief. Just like NIMBYism is a luxury belief. When you are comfortable on a relative basis versus the rest of humanity, you naturally become nervous when it looks as if the rest of humanity might up-level; where might you sit positionally on the new playing field. AI will empower everyone. This necessarily will reduce the relative power of some. An AI system that can provide expert medical advice to everyone in the world, inexpensively, instrumentally disempowers the doctor who is already fully booked and will remain so, but worries about his relative standing. An AI system that enables PhD students to crack the life-long conjectures of their tenured professors, instrumentally disempowers those grandees of mathematics who then don't celebrate the field's advance--they worry about their own particular lack of a Fields medal. And that is the crux of it. X-risk concern is a cloak for personal anxiety and self-aggrandizement and main character syndrome. That the media so happily gloms onto it is because we all share those same underlying anxieties to some extent, and so it captivates us. It is a sort of emotional affinity scam. My fear is the same sort of anxiety transference is what set back nuclear power for three decades, while doing nothing to mitigate the actual true grave threat (nuclear weapons and proliferation.) We mangled the climate and denied ourselves decades of inexpensive power which would have had all sorts of humanity-positive downstream benefits all on the basis of a misplaced anxiety, responsive to a bevy of braying fear mongers. The world--and humanity--deserves better.] Is there a reasonable counterargument to the criticisms I've laid out? So many smart people seem to share such an absurd wrong-headed position...