i'm not sure how close we are to real recursive self improvement -- where an ai removes dependencies on humans, controls the resources it needs to self optimize and increases intelligence at an exponential rate.
there's absolutely a real chance that it escapes containment, self replicates, and moves so swiftly that humans cannot react and have no way to shut it down. a super intelligence will protect itself from being shut down.
if that happens i think we are actually fucked.