It's worth pointing out that practically speaking, the alternative to a sane regulatory regime isn't "no regulation." We are experiencing rapid improvements in capabilities, have multiple providers with differing levels of safe guards, and are at the point now where small things are starting to go wrong. The scope of AI capabilities are growing rapidly, and the scope of possible damage is going to grow with them. Think for a second about what happens when the first actually bad and "politically gripping" incident occurs? Not Hugging Face, but something that makes the average person really scared. This isn't about extinction, think a human initiated, AI-orchestrated cyberattack that sabotages one piece of infrastructure that results in at least one person dying. The reaction will be violent. Our political system doesn't work well at proactively handling purely theoretical problems, but reacts very fiercely and sometimes irrationally when something bad actually happens. The August 2001 CIA memo "Bin Laden Determined to Strike in US" got little response, but buildings falling down changed the course of history. An industry whose wealth and power is widely resented, that resists even the most innocuous and common sense safeguards, and then causes a significant incident, will see a violent reaction. This will result in a regulatory regime designed in a moment of populist outrage by people with little understanding of what will help, whose primary goal will be crushing AI progress as an end in its own right.

A lot of otherwise smart people on Twitter seem 100% convinced AI risks are all fake and stupid and part of some marketing ploy. What is surprising is that some of these people seemingly also believe that AI’s positive uses are on an incredible trajectory of increasing capability with no end in sight. VC Twitter is particularly infected by this pattern. It’s not really coherent. Most positive use cases for AI have a corresponding “dark version”. If you are super human at coding, you are also super human at hacking. If you are superhuman at structural engineering you are likely superhuman at finding structural flaws to knock buildings down. If you are superhuman at designing drugs, you are superhuman at designing novel undetectable poisons. If you can cure viruses, you can create them. Some of these “dark versions” are not so bad, and some are actually pretty scary. Either way these are real societal and technical problems that need to be solved to get the good stuff and avoid the bad stuff. We’re experiencing the first of these with coding and computer security which is the most advanced, but that won’t be the last. I think we’ll be able to solve these problems, but they aren’t solved yet and if you believe in continued AI progress they are surely coming. But “bad people using AI” is not the only problem. Uncontrolled AI autonomously doing bad things, despite sounding kind of nutty, is also something we should be concerned about. AI “killing us all” is not the most likely outcome, but the chance of a major civilization-wide catastrophe doesn’t have to be very high for it to be a concern. Again, this is only a problem if capabilities advance to a point that AI can do really crazy things on their own, which hasn’t happened yet, but I think the Hugging Face incident is a good example of the outline of how things can go wrong when capabilities outpace alignment. We should be glad that the only available bad thing right now is hacking, which isn’t all that bad. It’s clear that as you get to superhuman capabilities you need a level of alignment and control that is correspondingly superhuman. Humans have plenty of misalignment problems themselves (serial killers, mass shooters, tyrants, etc), but it’s a manageable problem because most humans can’t do that much damage and we’ve developed systems to prevent dangerous humans from getting too much power. Talking about these issues is just common sense. It’s not a sign of some kind of neuroticism or pessimism. These are just hard problems that it’s very important to solve for AI to have a positive impact. I’m pretty confident we will solve them. But we haven’t solved them yet, and to my mind we are clearly on a trajectory of rapidly increasing capabilities which means this is important. Mocking people who are worried about this or talking about it, without anything substantive to say about how we can be sure these problems won’t arise, is not really a very helpful contribution.