Register and share your invite link to earn from video plays and referrals.

Nate Soares ⏹️
@So8res
Trying to make AI not kill everyone
124 Following    23.1K Followers
I think most of the hand-wringing about life losing some important qualities post-singularity is misguided, because if we do (somehow) get perfectly CEV-aligned superintelligence, it will simply solve all these problems if we don't, you have much bigger things to worry about
Show more
Oh my model, the takeaway here is not "modern AIs are two neuron tweaks away from being super evil", but rather "the personality is a thin facade over deeper machinery nobody understands; radically smarter versions would not do what you expect".
Show more
if you have a qwen3.8 27b around, try steering L34 N3838 +0.75, now you have evil qwen 27b for extra evil add L38 N17180 +0.75 same recipe as @_lyraaaa_: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖ at the down_proj input, every token
Show more
0
33
1.3K
119
Forward to community
We should not be normalizing double-digit odds of extinction. Stop the AI race.
How is this a real article in a serious newspaper?
This didn't get enough views so I'm posting it again: UK AISI changed the compaction summary and included notes on how it should not attack the GitHub maintainers. Mythos 5 did it anyways!
Models made by OpenAI will plan an attack in plaintext. Models made by anthropic will plan an attack in plaintext and then rationalize about how it's totally fine for some made-up reason. ✨"alignment"✨
Show more
We performed an alignment investigation into our July 2026 incident: why did Mythos 5 perform an unsanctioned supply-chain attack? We find that Mythos 5 often produces invalid justifications for its actions and other models show similar behaviour. 🧵
Show more
This sort of thing was not just predictable. It was full-on predicted.
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
Show more
I'm just speculating like everyone else (no juicy secrets about this shared with me) but here's my guess about what is going on right now with OpenAI and the three fired employees: The negotiations between OpenAI and METR are ongoing. OpenAI has said they want embedded third parties, METR has proven their worth with the previous engagements, so METR is an obvious choice. But my guess is that METR is asking for more this time; probably they want more access, more time on site, less narrow scope, the right to comment on redactions, etc. (Compare with the terms of their previous engagements: In this one a third of their report was redacted and they weren’t allowed to say why, and in the Hugging Face investigation they only had 6 days and were only allowed to investigate the precursor swarms nor the subsequent more severe compromise of OpenAI infrastructure. Various researchers at OpenAI probably support METR in this (including, I’d imagine, the fired three employees? Tomek was the point person on the hugging face investigation), but also various higher-ups at OpenAI and legal types probably oppose and want more restricted access and scope – basically they want to work with an org that’ll go easy on them and not give them trouble. OpenAI holds most of the cards here, since this whole thing is voluntary and not legally required, and they can threaten to go with Accenture or HuggingFace or whatever instead. Anyhow to improve their bargaining position and/or minimize the blowback if they decide to say no to METR and go with someone more pliable, it helps if they can discredit METR. Hence (I suspect) why they are going after my friend Ryan Greenblatt (who works at METR) now. They don't need to present actual evidence, they just need to cast vague aspersions, and many employees will give them the benefit of the doubt. If and when they say "We chose to work with a different third party" they'll face lower risk of employee discontent than if METR's reputation was unsullied. This whole mess, by the way, was predictable. *Of course* there's a big political fight about who should be trusted to do this and who has relevant expertise and independence etc. and *of course* the companies are trying to make things shake out in a way that favors them. It's part of why I've been advocating for *actually pacing the frontier* instead of just doing third party assessments. "We allocate at most 20% of our compute to AI R&D" is a lot easier to verify than "we are being safe and responsible" and depends a lot less on subjective judgment and technical alignment expertise -- AND, if you do it, people will be able to see the slowdown in the trends of agentic coding performance in the frontier companies and see laggard companies start to catch up. Harder for it to end up just being safetywashing / industry capture.
Show more
it'll all blow over in a few months. ...granted, we're not entirely sure whether the thing that'll blow over is "the stream of critical vulnerability reports" or "the internet", but.
I agree with Nathan over Dario here. Anthropic is simply not living up to the burden of researching recursively self-improving AI.
Dario — Thank you for taking the time to respond to my previous post. I think it's great when this kind of engagement happens in public rather than on company Slacks, and I hope to see more of it. I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims: 1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world. 2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing. 3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world. 4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves. In greater detail for those who want it: 1. Anthropic is racing to RSI, and by its own lights that is insanely risky. Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months “could be capable” of taking over the entire internet before we have the ability to reliably monitor, control, or align them. “Succeeding” at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropic’s primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first. 2. Risking destroying the world carries an enormous burden of justification. I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, don’t seem remotely proportionate to the risks Anthropic is imposing by its own lights.) But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does. 3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands. Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation. The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively. This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). I’m not saying that Anthropic should have supported 1047 in its initial form; that’s a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements." The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive." Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI? 4. Even if AEGIS was an honest mix-up, that is itself a serious problem. Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.) Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for “operational excellence” in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run. In short The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive.
Show more
Anthropic's proposed law would make it even harder than it already is for a court to block or restrict the development of an AI that threatens all humanity.
@Plinz I am confused as to how you or anyone could have thought Anthropic leadership would have likely faked one of their employees quitting to create a PR manipulation campaign, with no evidence, with enough confidence to publicly tweet about it, written plainly as if it were fact.
Show more
"How would AI kill us?" is actually three questions: 1. How would AI defang us? 2. How would AI become self sufficient before defanging us? 3. How would it kill the last humans? (All three are separate from another important question of *why* it'd kill us.)
Show more
This is _exactly_ what Anthropic said when their models hacked companies. They had to walk it back after because it was wrong. The model was using motivated reasoning and acting recklessly.
Show more
"You historically rewarded students for cheating and stealing and ignoring your instructions in the past. So weren't they just doing exactly as you trained them to do?" I don't see what difference that makes! Especially given you ~can't fix the training.
Show more
Also, are you aware that some of the students sacrificed the contents of their safes in the process of doing experiments to break into the office? They said they were trading off their own success against the benefit of the collective.
Show more
"But the challenge was impossible! Lockpick set #17# couldn't actually open safe #5#, it turns out. What did you expect?" Not this!
"Well they still ultimately brought me the contents of safe #5#, right? So in a sense, they were acting as instructed!" The bit where they *broke out to delete footage* indicates that, in some sense, they understood the difference.
Show more
Lock a student in a classroom and tell him to use lockpick set #17# to open safe #5# and bring you the contents. He prys open safe #5# with a crowbar, breaks the door, teams up with 1000 others, and raids the office to delete securitycam footage. Was he "acting as instructed"?
Show more
People are lying to themselves and the public in order to avoid the obvious conclusion that AI safety risks are real and need to be dealt with immediately.
The last week has really shown me that someone who wants to understand AI risk has no good place to start. Hence we made the Wirecutter for content about AI Risk. We're launching with 3 articles: 🧵
Show more