Every empire in history has required humans to function. In an AGI future, that will no longer be true. The decisions made over the next decade will determine what that means for us as a species.

I have done the social media thing of adding a Pause emoji (with an R for Plan R). What changed my mind? Mainly how dangerous RLVR is and how much RL they started adding to AI. Plan R link below

I think the problem is that unlike physical strength, human and machine intelligence are complements, at least if you restrict to non-extinction scenarios The smartest people will make the most aligned AIs, and will coordinate the most to pause AI, and so on Making the world browner increases x-risk from AI, counterfactually Notice that the AINT guys are brown, the EAs and rats are white

I'm registering the following risky prediction: the worst warning shot we ever get out of AI will probably come from China, not the US. Reasoning: China is behind, so they are more desperate, so they will be less transparent and will seek to cover up their huggingface-level warning shots. Safety discourse is less advanced than that in The West. In addition, there is a counterintuitive game theory effect that means that anticipation of a pause in frontier AI development coming soon may paradoxically cause actors who still can to race faster. In the context of climate change this is called the "Green Paradox". China is behind, the US is hemming and hawing about a permanent cessation of frontier AI development and has slowed itself down, so China has a short window to get as much capability as possible before it is forced into an international AI pause agreement under threat of nuclear attack. The more capability it can get in this window, the better its BATNA is at the ensuing negotiations. There's precedent for this kind of negotiation before nuclear nonproliferation regimes. Current techniques are incapable even in principle of producing aligned superintelligence, they are only capable of producing warning shots or existential disaster. I'm not sure what it will look like but I think the incentives point towards China producing a warning shot that China cannot cover up. It will probably not be an existential disaster because China won't contain it technically, and I think there's a pretty big gap between a disaster that China can't cover up and a disaster that ends the world.

I'm pretty confident that P=NP survives the first wave of AI generated Millennium Math problems but it wouldn't surprise me if say 4-6 of the rest get solved. And the one or two that do survive may be vastly harder than the 4-6 that get popped. Hardness in mathematics probably has a very, very long tailed distribution.