Wrong understanding -> wrong action -> wrong outcome. Belief that you can predict the end of the world and that we need to create policies to stop an imagined apocalypse can and do cause real problems. The Population Bomb led to the One Child Policy and caused the very problem it was trying to solve. Anti-nuclear activists fought one of the cleanest energies on the planet to 'save the planet' and ended up doing tremendous damage to the planet they were trying to save (they are still doing it). Mao's Red Guards believed that capitalism was evil and that only collectivism could save China and they helped cause massive starvation and famine across the countryside. History is chock-full with people promising to save us all from the end of the world and doing massive damage instead. Beware of do-gooders baring gifts.

Do we ever get to track actual harms in reality or do we just track imaginary future harms? Because with 362 tracked incidents of LLMs in 2025 and 1.2 billion users that's a rate of about 0.000030%. Maybe safest products in history. Definitely time to act now to protect us all

When we in the AI community say that the narrative of "rogue agents" is premeditated and pre-crafted and mendacious and where the evidence is twisted to fit that narrative we mean it literally: Buck Shlegeris, CEO of RedWood, one of the two firms pushing this narrative, wrote an essay in 2024 ( called “Would catching your AIs trying to escape convince AI developers to slow down or undeploy?” where he wrote, “It might be worth having a plan in place for how you’ll persuade people to freak out if you actually run into this evidence.” Chris Painter, president of METR, ( just wrote, unironically on X: "Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to 'going rogue,' the public would find out...This is what we've been focused on since 2022." In other words, the focus of these organizations is to tell this story. Actual cybersecurity and cyberdefense is not and never will be the focus. Their purpose it to post scary white papers, go on podcasts and TV and terrify the public. Their goal is to slow down or stop AI development, by any means necessary.

When you have a magical constant that you can inject into any equation to make the equation magically work, like Superintelligence, then you can convince yourself of anything. It's similar to extreme religious arguments. Q: "Why are you doing that?" A: "Well the book told me to." Q: "Well who wrote the book?" A: "God." Q: "Well did he really or was it just some ordinary people?" A: "Divinely inspired people." Q: "Okay but like how did you know that?" A "Because the book told me so." And round and round you go.

Launch day! Introducing Nautilo, the open org harness for everyone. 100% open source. Server. Desktop. Web. Android. iPhone. Every user gets their own Genie, a powerful and radically loyal super agent. Think the flip phone version of Her, but for teams. She's a friend and secret keeper. She remembers everything about you and never breaks character. Not an agent. A machine person. With you for life. And because you control the server, the models and the end points, you can trust her with anything. Nautilo is self-sovereign and private. Our first service pack next month will bring full end-to-end encryption. Genies get only short term delegations to decrypt. The desktop app comes stocked with apps for you and your Genie: A built in web browser, a cloud browser, a full office suite (docs/slides/sheets/Miro style board), a full video editor, a vector designer, a terminal, and built-in Google Suite (after a few Google hoops). But unlike other platforms these apps are co-creative. In most other systems only your agent can edit. It's prompt and pray, hoping she gets it right on iteration 42. With Nautilo, you can take over when AGI doesn't get it right. She's also got built in third party coding harness controls that she steers like you would: Codex, Hermes, or Claude Code. Call them when you need a specialist or some cheaper compute. Make your Genie today. Carry her with you for life. Site: GitHub: Why open source all this? Because we believe that if your org level harness isn't fully open you can't trust it with your real work and your personal world. A closed harness is just a surveillance app in sheep's clothing that a SaaS or model company can rip away from you overnight. Control the harness and you control your future. Code is beta but getting stronger by the day. And with that, here's a mega-thread with video deep dives:

Uncle Bob breaking down the classic Doomsday playbook of "insert magic into the equation to make the equation work." It goes like this: Doomer: I've just given you a realistic scenario of the end times, you just won't admit it! Skeptic: I mean, I read it. So your premise is seriously that nano machines kill us all? Like you know those don't exist right? And neither do automated virus labs, etc.. Doomer: But what if they DID exist and I inserted this magic non-existent thing into my equation, would my equation work then? Skeptic: Well yeah but... Doomer: Ah ha, so you admit it! Case closed. If we had nano machines and automated viral labs we would all diiieeeee... Skeptic: No what I meant was obviously if you insert a magical variable into an equation it works but that's not realistic... Doomer: Case closed. We're all going to die!!!! I'm not going to save for retirement anymore and do lots of drugs.

One thing will defeat the AI Doomers easily: Time. Little by little things will change and AI will diffuse into everyday people's lives and they'll realize it wasn't the scary bedtime story Harry Potter told them. Soon enough everyone has a friendly AI personal assistant that knows everything about them and that they *love.* Not long after, a self-driving car saves a friend's life from certain death by swerving with super human reflexes to avoid a horrible accident. In another small town, personalized AI medicine saves someone's grandma from cancer. In another, a kind and gentle robot takes care of a mother in her old age. And then the mood shifts and people realize what these folks are trying to take away from them *personally.* And pretty soon passing restrictions on AI is like passing benefits cuts on senior citizens. And the more they push, the more everyday Americans will realize they're trying to take something away that they love and they will be very very mad. And that's when this Luddite movement goes the way of the last one.

What's happening right now with AI existential madness: "The people can always be brought to the bidding of the leaders. That is easy. All you have to do is tell them they are being attacked and denounce pacifists for lack of patriotism and exposing the country to danger. It works the same in any country." - Herman Göring

If we're doomed to get the Oversight Committee for Pacing Economic Innovation and Development, we need to make sure it's not stacked with the drinking buddies of the folks stumping for it. METR is not an independent evaluator of anything. We need real cybersecurity engineers.

These are not serious people. Each of these is b tier sci-fi that requires additional made up, non-existent, imaginary stuff. Luckily all the super magic machines need are nano machines to kill us all! These people need to stop smoking crack and go back on their meds.

TLDR (written by AI because I can't be bothered to write short pieces for short attention spans. Original piece written with my own two thumbs): Today’s frontier models can’t “copy themselves like a virus.” They require terabytes of weights, specialized inference software, tightly configured clusters, and millions of dollars’ worth of heavily monitored GPUs. Hijacking that infrastructure would create enormous billing, performance, and operational anomalies and displace whatever valuable workload was already running there. In no way is that not noticed. Could this become possible as models shrink and compute becomes more widely distributed? Sure. But today it’s a foreseeable, mitigable engineering problem, not spontaneous digital reproduction. This propaganda campaign assumes AI advances while infrastructure, monitoring, security, and economic incentives remain frozen in place. That’s not how technology evolves. That’s sci-fi with zero understanding of the real world or actual DevOps.

Tell me you have zero DevOps skills without telling me you got zero DevOps skills. And LLM is not the I Love You Virus. It is not the same as infecting a web server. It is not the same as hacking a wiki. One is a commodity piece of software that is widely distributed and this capability is a non-trivial leap in hardware/software/capabilities and one does not automatically follow the other. Currently, it is a massive effort to run these models, usually distributed across multiple very expensive chips, running on specialized inference software, specialized harness software, databases, cloud computers, with tooling and hundreds of install packages, etc. Kimi K3 is 1.6 TB of space at 2 trillion parameters and you need about 1.5 TB of VRAM, which means you need 8xGB300s which will cost about 350-450K a year to purchase or as much or more to rent. Now let's say Astra or Mythos or later models come in at 10 trillion parameters, so that puts us at roughly 10 to 20 TBs unquanticized. We are guessing her because these folks reveal nothing about architectures anymore, but they are reasonable guesses based on scaling laws. You would need roughly 23 TB of VRAM to run it and roughly several million dollars to run that one instance a year. And to make it worse, it no longer fits on an 8 way cluster, which means you now need special networking configuration and inference software to run it or you need to infect a Nvidia NVL72 rack-scale server system costs between $2.8 million and $6.5 million. So the "rogue" model copies itself to the most expensive compute in the world. It's safe to say that hardware is not sitting idle and is being watched closely for downtime or anomalies. In fact, it is likely doing one of the following things: 1) Serving customers at a hyperscaler 2) Serving inference for a well endowed company. In the first case it will be instantly noticed by the actually good automated monitoring systems of the hyperscaler because there is now something using their infrastructure but not being billed properly and costing them millions of dollars in lost revenue and electricity. In the second example, the model they were serving on that very expensive hardware is now offline (replaced by the rogue model). The model presumably was connected to a valuable piece of internal software that is now glitching and no longer running. Nobody notices this? And so we see that this is where sci-fi meets the little problem of the real world where friction, dust, time, resources limitations and the like come to crush your little fantasies. To be fair, this *may* be possible in coming years. I assign it a non-zero probability. Breakthroughs in new architecture and smaller models and eventually the wider distribution of the computing substrate that commoditizes could change this materially. But for now, it is mitigable and foreseeable and an engineering problem. In no way will these things be copying themselves around like a virus on your computer in their current form. It also betrays a lack of understanding of the evolution of complex economic and societal and technological systems, we tend to get mitigations as parallel evolutionary developments. Things do NOT evolve in a vacuum. So when you get viruses, you get anti-virus software. When you get DDOS attacks, you get CloudFlare. There is money in solving problems and things do not evolve in isolation. Believing that things evolve in isolation, or that one thing changes and all other variables stay the same, is one of the many reasoning errors that destroy people's ability to make accurate predictions. In addition, when you don't have the requisite knowledge to verify your predictions, things that sound sane and rational to you are, in fact, insane and ridiculous.

You cannot do serious research about an imaginary form of a technology that does not exist. If you wanted to study the propulsion of disc based UFOs that bend gravity where would you start? You can't start anywhere because it does not exist. And it follows that these fears do not exist in reality either and are based on nothing.

It's time to call these end-of-the-world anti-AI messaging groups exactly what they are: Terror cells - They funnel and wash money from a few donors to smaller groups to obscure origin - insert sleeper cells into orgs (the best recent example is Jacob, the young man who quit yesterday and created a social media firestorm, has practically no resume and took 20K from these groups while working at Anthropic. He quit this company, which is probably the most dedicated safety AI group in the world, while calling it "dangerous" with a straight face, for a grand total two month, had no social following and suddenly gets retweeted to 100M views by various cells working together behind the scenes to pump and promote and manipulate the public and regulators.) - these groups produce streams of propaganda designed to terrify the populace, such as reports, push polls, articles in magazines and newspapers, podcasts, as well as direct funding of videos deliberately designed to terrorize children about AI in YouTube - pay influencers to produce anti-AI terror messaging - Inspire and produce splinter cells, like the Zizians, a cult splinter group from the San Francisco Bay Area rationalist community, that committed a double homicide in Pennsylvania, knife attacked several people in California, and a got in a shootout with a U.S. Border Patrol agent in Vermont - they openly call for violence on social media and in private groups with calls for "Butelarian Jihad" a reference to Dune which took its inspiration from the violent Arab uprisings and holy wars - act quasi-independely but coordinate attacks - use timed campaigns to manipulate social media - one of its members threw a firebomb at Sam Altman's house in an attempted murder - another shot up a judge's house because he voted in favor of a data center - several of its members call openly for more violence on social media In a time when many people are looking at these folks as if they're exposing real risk instead of what it actually is, which is a delusional, nonsensical, imaginary risk, it's time to get clear about where these fake risks are coming from and why. The burden of proof for proving that a technology currently used by billions of people happily, with practically no serious incidents (by comparison lawn mowers are a bigger menace to society), should be on them. They need to provide more than bald and bold assertions with no supporting evidence for their claims. It should be very clear that this terror campaign is not organic and if you're an actually rational and clear thinking person, it should make you seriously question the origin and nature of these groups and what they have to say and the policies they propose. We must not let terror cells corrupt and manipulate the American public and legal system with coordinated terror campaigns.

Not only I am tired of these wild AI speculations of impeding doom from self important people: I resent them. I actively resent people proposing to crash the economy or proposing authoritarian control over my life and other people's lives with idiotic and dangerous ideas like chip control or bans or tracking researchers. Every idiot in history who's taken the approach of "the ends justify the means" to solve an imaginary future disaster created the very disaster they wanted to stop. See Population Bomb leading to the one child policy and Communism leading to Mao's mass famines and Fascism leading to the death of tens of millions of people and war. These solutions are evil and worse than any disease that they propose to solve. It you believe you can actually predict the end of the world then you are as insane as the Heaven's Gate cult that killed themselves in the 90s thinking UFOs were coming to transcend them. Not only should we not take your policies and fear mongering seriously, we should actively throw out any and all of your proposed solutions because they come from a place of delusion. I don't give a shit that you work in the industry or think you saw something or that you want to virtue signal on X. You are actively contributing to a horrible future based on baseless speculation that has no grounding in reality. Don't confuse expertise in a domain with ability to predict impact of that domain or frankly to make predictions a decade out better than a dart throwing monkey. They are orthagonal skills. How's Hinton's "we don't need to train Radiologists anymore" working out? How did the Population Bomb work out? The global cooling second ice age? Peak oil? Just because someone is a bridge engineer does not mean they can predict the impact of bridges on society or that they have any actually useful insight at all on the complex, ever changing system called life. There are people dying in wars right now. Children starving. Homelessness. Dictatorships. In short, real problems. And we're supposed to drop everything to stop a made up problem in your head? Pound sand. We don't care. And we are not going to remake society based on you scary monsters under the bed delusion.

AI doomers are delusionally dangerous. Their solutions to an imaginary problem are: 1) Rigid authoritarian control over our lives and AI Or 2) Crash the economy by bringing progress to a grinding halt Again, all to solve a completely imaginary problem.

Here's a few reasons why people have so much trouble predicting the future: 1) Fail to realize that even the best superforecasters' predictions drop off dramatically past a three year time horizon and that most folks are worse than dart throwing monkeys at future predictions. 2) Fail to realize you literally can not see black swan inventions coming around the corner. If you predict the future of Germany in 1439, then in 1440 your predictions are completely wrong because of the Printing Press. If you're an 18th century farmer you can't see a web developer job because it exists on the back of countless developments and inventions you can't predict. 3) They change one variable and hold all other variables the same. i.e. AI advances and nothing else does, no parallel discoveries or innovation, no solutions, no mitigations, no societal or cultural changes. 4) Predict unlimited resources and zero friction in the real world (dust, disconnects, diffusion, etc) to slow/divert/change/impact the development. All changes experience equal and opposite reactions. 5) They mistake their ability/expertise in a domain for a parallel/orthogonal ability to predict the future of that domain and its impact on the world. Two different skills and they do not usually overlap (though very rarely they do.) 6) What I call "classic sci-fi or Jules Verne syndrome", which is similar to one variable changes. It's like in Jules Verne when one guy gets the submarine and nobody else does. But life is more like cell phones, lots of people getting them over time in a diffusion curve. 7) They mistake exponential curves as infinite always and never see an S curve coming. 8) The see infinite resources (compute/memory/learning upper limits/improvement) and no limitations.

Hinton finally admits he was wrong about radiologists because he didn't understand: 1) Jevon's Paradox (when something is easier/cheaper we use more of it) 2) The job of radiologists Maybe people who are gifted in a domain are not usually the best people to predict the impact of that domain on the world (because that's a totally different skill)? And maybe, just maybe, he is wrong about the big risks of AI too? Now remove the "maybe" and you got it!

We'll have some real problems with AI. What technology did we ever create that didn't have some downsides? Road crashes kill nearly 1.2 million people a year and injure 20–50 million more. But people still get in their cars every morning, grab a coffee and drive to work. They cross bridges that could collapse, live in houses that could catch fire and climb into metal tubes that fly them across the ocean but could burn up and break apart. Somehow people manage to get through breakfast without demanding a mathematical proof that nothing bad will ever happen. But AI? Pound for pound, AI is one of the safest technologies ever deployed. Safer than lawn mowers by a massive margin. And yet it causes more deranged and apocalyptic thinking than any other tech in history, with very serious people tell you with a straight face that AI will take over and go HAL with 100% certainty, based on no actual real world evidence whatsoever in the real word except some agents posting to message boards and hacking and that is extrapolated into the Population Bomb. And we've got people screaming that we need to anticipate every possible failure, every misuse, every terrible thing anyone might ever do with it before we're allowed to move forward and make sure it really, really can't possibly go wrong. It's ridiculous. It's worse than that, it's a societal level mass hysteria. Just look at the stats: Waymo reports 94% fewer serious injury or worse crashes than its humans across 220 million autonomous miles. That means self-driving cars are much safer. To the point that we'll probably see human driving made illegal without a special permit within twenty years, maybe faster. But how is that covered? Every single crash is picked apart and screamed about in the news the same way we pick apart the top athletes in the world when they suddenly make a single mistake and loose a match. We take something as close to perfection as is possible in the world and ask it to be even more perfect. We used to have a much better understanding of risk versus reward in history. We accepted that life was uncertain and that things could go wrong but we dared to dream and do things anyway. A generation that demands perfection before we even get out of the gate is a generation who'll watch their children grow up poorer, with less opportunities, as their civilization gets out-paced, out-matched and out-competed by more daring and bold civilizations that blow past them while they're dreaming about a childproof world that will never be.