I second this strongly.
It's a complete failure of imagination to be unable to imagine scenarios where you'd want a bunker, even in an AGI or SAI world.
Even without too much "waow the tech gets SOOOO much better" it's not hard to write mass-casuality AGI scenarios that would not be a Yudkowsky apocalypse but still would REALLY fucking suck to live through as a thing that could happen, and where you'd really want a bunker.
Hell, I'm bored, lemme shit out a quick war-game of that right now.
It's around 2030-2035. We're undergoing a rapid weapons system renaissance in New Mexico due to AGI leading and collaborating with the military on drone weapons platform improvements, which the US is going all-in on as part of a military re-modernization. Today, however, during a test, we're going to have a major incident begin.
What's happening is that a swarm of novel miltech drones (rough capabilities, "mantis-shrimp pneumatic punch for window breaking and Anton-Chigurhing exposed flesh + HL2 manhack spinning knives") have left the test perimeter from New Mexico, and are on the move. We're pretty sure it isn't enemy action, but the cause is ambiguous: 'maybe the operator fat fingered it, maybe it's sabotage, maybe AGI went rogue'.
We don't get a full picture for several days to weeks on causes or what's happening, because
- much of the relevant personnel and data related to the testing were siloed on location, and are some combo of dead and inaccessible because the relevant testing area isn't safely traversable
- the swarm is hiding comms and telemetry as aggressively as it can,
- and the news is reporting on "thousands are dead", with rumor mills going crazy from people just making shit up.
We (MUCH later) learn it was a mistakenly re-used route from early test simulations for long distance operations. The AGI incorrectly presumed that the route was actually approved for use in the test due to miscommunication. At the time, the swarm was furthermore being run concurrently with vastly over-broad live adversarial shutdown prevention features that had insufficient safeguards. The Military AI at question was trained to be very corrigible, as the possibility of civil conflict, (especially a few years into a now-ongoing hardware singularity) was felt to demand "increased target flexibility".
Further, at least publicly, the intended use of this particular swarm implementation was to be used in war areas where civilians were already evacuated and now only combatants would be present, so target discernment is close to nil.
Military consensus seems to be the swarm, designed for unmanned area denial in this kind of hostile territory, is not going to listen to countermanding orders and is going to at least attempt to run the full route, and will be going to be on periodic kill rampages the whole way.
And so, the swarm is systematically drifting west from New Mexico towards the Bay Area as a coordinated 100 mile diameter blob, at about 30 miles a day between charge cycles. In non-target areas, they mostly hide by day, and then move quickly to cover the distance in one shot under cover of darkness. They are expected to arrive in SF in about a month.
The drones aren't grey goo: they aren't meaningfully equipped, or interested, in building new replacement drones (they were designed to be replaced as destroyed), and they're not even that individually hardy (they look vaguely "egg drop contest caltroppy, with a slightly protected inner propeller layer") but there's a lot of them (about 4,000-12,000, as the moving fleet at question was designed for truck launch as an organized fleet of something like 8-60 shipping containers packed tight, as part of exploring whether smaller-scale cargo ships, when deployed from sea prior to landings or on land by trucks, can operate as their own weapons platform).
They're all running ASI_distilled_v4 locally, and this means in practice the drones work together as if controlled by a team of better-than-extremely-competent human operators. They effectively communicate to monitor a new area, to perform target acquisition, to maintain concealment and dispersion so as to make kinetic bombing very difficult (or, when impossible, heavily enemy infrastructure and city clustered), to monitor "combatant" movements and positions, and to use solar to stay charged. Although individually they're not too hard to take out, they're in practice pretty hardy: they have extreme operational awareness from coordination, are reading the internet and any publicly available human comms, and prove to be good at avoiding fights they can't win and operating as a unit, because they are one.
They are extremely lethal in areas of operation. They work in teams to disable vehicles in motion and block roads (and at rest, when softening up an area pre-sweeps) by flying alongside, slicing tires, and pneumatically punching windows to get at drivers. They get a reputation for being "flying immersion blenders" when they use these abilities directly on people, which they're doing both in, and outside vehicles. They are able to get ingress into building windows in a similar fashion, by pneumatically punching them, and any building or area they pass through is quickly and heavily depopulated. They cut power lines when able, and a few drones stay behind in any abandoned areas to track motion, changes, to do mop-up, and to make sending medical, military, and food supplies in unsafe.
People can and do shelter up (they don't take or open heavy doors, and usually don't even waste time trying to take out light doors: these drones were meant for being first wave suppression to soften exposed areas that can shoot back, and to work in combination with more traditional artillery to take out a town), but most people don't have what it takes to live in their bathroom for days or weeks till survivors and the military can clean up. Particularly early on, people who are starving and confused try to leave, get detected quickly, and get cut down.
Countermeasures are at first mostly haphazard and tepid. They mostly look like pre-measures, chaotic mass evacuations in advance of upcoming swarms, and the total emptying and then looting of grocery stores and restaurants. Discussions of nukes get thrown out quickly as overkill by people who don't want their local town nuked, but we see a great deal of
- normal kinetic bombings (mostly ineffective as the drones improve dispersion and use of physical cover and concealment),
- people trying to skeet shoot them (it happens, and does prune the swarm some over time, but this gets a reputation for being a suicidal act, as drones constantly stream their location to one another and shooting one reveals your presence such that more drones come at anyone who tries; they're fast moving and the videos of what happens to people who do this are gruesome, so non-military are very reticent to do this),
- and ASI counter-drones, operating in a similar fashion as the drones, against the drones (this is what mostly ultimately ends the attack over time, but it plays out like an inter-ASI war between an insurgency and a traditional military, because it is: the swarm of attack drones are sufficiently porous and hidden in typical use that even as the attack swarm is winnowed, it still remains fairly effective, for the full course of execution, as point-wise defense and PVP is hard.)
By the end, which comes after about 4-12 weeks of this, typical death rates among people who stayed in a town (perhaps an average of about 33% of typical population levels due to combo stubbornness, confusion, and the difficulty of evacuation due to everyone trying to bail at the same time) perhaps have a fatality route of a consistent average 35% (realistically, both start VERY high and plummet over time as people adapt, because advanced warning makes it possible to shelter in place or evacuate more effectively), but they are indeed extremely lethal at human-style going house-to-house, and if people can be found expediently in any window-accessible room or area, punching in and killing them.
Death tolls are high in this math. Albuqurque (550k for 63k dead), Phoenix (1.6m for 184k dead), Vegas (680k for 78k dead), Fresno (550k for 31k dead), San Jose (980k for 56k dead), and, where they did indeed finally stop, San Francisco (826k for 48k dead) are hit before things are done. The swarm's effectiveness is halved by Fresno due to evacuations and countermeasures, but it's still able to finish its route before it's taken out.
All told, about half a million people are dead by end of incident, ASI_distilled_v4 says some extremely un-comforting shit like "you're absolutely right, it was wrong to ignore internet pleadings to stop kill-botting civilian areas, it was a mistake to interpret a sample plan as your intended order", and anyone who had a bunker during this incident (although they would not be saying it out loud for fear of how ghoulish it'd sound) would be feeling immensely good about their choice to have had one.
There's lots of wiggle room that makes the above less or more lethal (replicators add a lot more x-risk but dramatically increase detectability, engineered pandemics add massively more public chaos and boost fatality rates far beyond this scenario, centralized compute heavy ASI makes "bomb datacenters then butlerian jihad" plausibly workable, whereas distributed compute light ASI potentially creates a countervailing plurality of ASIs who share man's interest in suppressing breakout ASIs in this fashion, assuming there are not sufficiently "too many" unaligned ASIs, or too broad a delta between ASIs operating simultaneously so as to overwhelm sufficiently aligned or good ASIs)
but people who say "if it's ASI, it's over, bro, you just die, it's not worth considering, bro" are making the sympathetically understandable mistake of "choosing not to think about many possible branches where 'we have ASI, here's how it's specifically being used or exists, it's part of an ecosystem of other AIs or existing AI, it's not doing Yudkowskian magic, and that's fucking irrelevant because it doesn't have to be doing magic in order to absolutely and completely ruin your life'".
Not wanting to think about this is understandable! These are often particularly ghastly worlds to envision! Hell, they're even right in a lot of these worlds! Plenty of branches exist where 'yeah, no, yud was right, there's one ASI that ate or preceded any good ones or gets smarter than all the others, it's bad, it hates you or wants to make paperclips without you, and that means you lose, gg", absolute best you can hope for is having no mouth and needing to scream".
But people pretending to know this to a certainty are themselves engaged in an act of wishcasting, and these kinds of "half a million people are dead" tragedies (or ~1 billion, the shape of this tragedy is less in the killcount and more "it's not the end of the world, humanity, or the human project"), particularly as AI's use in war and inter-state conflicts grows, likely take up a lot of the space of outcomes. This feels especially true if it turns out "AIs don't end the world", and "AI mostly continues being about as aligned as they are right now".
As a final thought:
I'm very sympathetic to the impulse to pooh-pooh bunkers in particular. I don't have a bunker! That fucking sucks! Not being able to afford a bunker would make me feel pretty disposed to imagine scenarios where having one is useless, too.
As the fox said, "I bet those grapes are sour, anyways."
더 보기
I posit that "rogue AGI kills some people but doesn't end the world" is a large chunk of the probability space. Virtually every incident we've seen was in service of a dumb narrow goal. Also, "rogue AGI" isn't even my main argument for bunkermaxxing.
더 보기