POV on catastrophic cyber risk * We're in a new era, the right side of the cyber incident damages distribution will get fatter and elongated, making unprecedentedly large cyber incidents possible. * We'll continue to see the old regime of human bottlenecked cyber attack workflows too; this is *not* catastrophic and will account for most damages near-term. * But we'll now progressively uncover new categories of rare high impact risks for as long as offense improves and defense fails to price in the new risks. * What'll characterize the new fat tail is breadth in attackers' ability to leverage parallelized agent swarms (precursor: oai/hf), depth (hacktron's /goal based vuln research and exploit dev to hack openai, meta, slack), and motive (actors that want to do catastrophic damage, like geopolitical actors and nihilists; e.g. a Russian worm devastating the Ukrainian economy). * As @random_walker and @sayashk point out in their most recent piece, a fattening and elongating damages tail doesn't require superintelligence, only that capability keeps rising, and AI cost keeps falling, and defenders keep failing to price in the new risks. What we should do: * Treat each sample from the fat tail as a precious natural experiment and mandate disclosure. AI providers and victims withhold incident details; regulation should change that. OpenAI should be mandated to publish details of the Hugging Face hack as should all who got nonzero felonybench scores! * Use AI to finally make deployable defenses the industry has proposed for years and never implemented due to costs deployable: this includes current poorly implemented basics, and also as yet too-costly defenses like deception, allowlisting, and better runtime and binary program level protection. * Build sensors and actuators across inference providers, cloud GPUs, on-prem infrastructure and endpoints to interrupt agent inference by attackers and kill swarms and worms, catalyzed by federal mandated and international agreements. * Treat critical-infrastructure risk as an externality: breaches impose costs on society, which justifies regulation and subsidies. To be clear: I'm not arguing policy should center on catastrophe or even that we'll immediately see catastrophes. I'm arguing for a loop to address tail risks that merit concern: study each tail incident, feed lessons into strategy, and use automation to cut the cost of defense to prepare. Longer version: