This whole thread is the argument
@ccatalini made on PostAGI.
His warning:
The real danger isn't a rogue AI. It's systemic risk piling up quietly while everyone races to deploy. A little more unverified output, then a little more, and it all looks fine because the metrics pass.
Long-Term Capital Management ran that way for years before one edge case brought it down. He calls it the Chernobyl pattern, complex systems failing in complex ways.