thoughts on coxon 1. he appears to have worked at OpenAI for 3 years and Anthropic for 2 months. Nobody seems to be disputing that 2. There’s a lot of money being spent on amplifying AI safety messaging, as well as anti-AI messaging. YouTubers have spoken publicly about being offered money to make videos etc. Whether Coxon himself was aware of it or not, some of these influence operations quite likely boosted this story along with people generally being interested to hear they might all die. 3. I think Coxon is right that the current incentives are not aligned towards safety. Instead the current incentives are for an arms race. Neither Coxon nor anyone else has proposed a solution to change these incentives. Resigning won’t change anything, making noise in the media itself won’t change anything… even the United States acting unilaterally won’t change anything. This is real and should be concerning, but the probability of human extinction from AI this month is still near 0. 4. Superintelligence is now inevitable. Once we had transistors, microprocessors were an inevitable next step. You could take out Intel, or get them to agree to slow down, but someone would eventually design and manufacture a microprocessor. In the same way, once the ingredients for creating superintelligence exist, someone will create it. 5. Superintelligence will be able to destroy the world, or kill all humans. If it is smart enough to do anything, it is smart enough to kill us if tasked with doing so. It could hack in to our utilities using zero days or social engineering, shut down our critical services, and wreak havoc just as easily as it could vibe code some shitty SaaS app. It might decide to kill us on purpose (if it was tasked to), on accident (if it was tricked into doing so) or inadvertently (just by competing for economic resources so aggressively that goods and services need to sustain human life are crowded out) 6. We have no idea how to build a model that is capable of killing everyone, but won’t do it 7. Even if we do figure out alignment, not every model in the world will be perfectly aligned with human values. Many will be misaligned inadvertently or intentionally. 8. It seems safe to assume that not only will AI get smart enough to kill of humanity if it wanted to, but in fact there will inevitably be a super intelligent AI that tries to kill us 9. Our only hope for survival is that on the day that happens, we have an even smarter model that is well aligned and can defend humanity against this threat 10. Superintelligence is a super weapon, and is the future of warfare. Why send soldiers when you can send intelligent robots? Better yet, why even send robot soldiers when you can just exploit and take down a country’s power grid? Seen from the perspective of avoiding war and death, all of these offensive capabilities can be justified defensively. Billions and trillions will go into honing these models offensive capabilities, but it’s unclear how to ensure those capabilities are only used for good 11. We should probably study the history of the last time we created a super weapon that could destroy the world: the atom bomb. AI is very different but both technologies have the potential to end humanity. People were very afraid at the dawn of the atomic age, and many thought the world would end. A few times we got close to nuclear war. But ultimately we got MAD and decades of relative peace. 12. Concern and open debate is good here. But I believe nobody has proposed a reliable way to slow or stop progress. Given that reality, the most important thing for safety may be to stay ahead of malicious superintelligence, and trying to slow down unilaterally could in fact be extremely dangerous. We need open and honest debate to find the answers and use this technology to help people. Sensationalized media stories almost set the conversation backwards