I think the problem is that unlike physical strength, human and machine intelligence are complements, at least if you restrict to non-extinction scenarios
The smartest people will make the most aligned AIs, and will coordinate the most to pause AI, and so on
Making the world browner increases x-risk from AI, counterfactually
Notice that the AINT guys are brown, the EAs and rats are white