this tweet from a former OAI researcher seems harder to argue with every day that goes by security is only as strong as the weakest link and these AI companies software stacks have so many links

Dario — Thank you for taking the time to respond to my previous post. I think it's great when this kind of engagement happens in public rather than on company Slacks, and I hope to see more of it. I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims: 1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world. 2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing. 3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world. 4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves. In greater detail for those who want it: 1. Anthropic is racing to RSI, and by its own lights that is insanely risky. Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months “could be capable” of taking over the entire internet before we have the ability to reliably monitor, control, or align them. “Succeeding” at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropic’s primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first. 2. Risking destroying the world carries an enormous burden of justification. I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, don’t seem remotely proportionate to the risks Anthropic is imposing by its own lights.) But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does. 3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands. Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation. The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively. This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). I’m not saying that Anthropic should have supported 1047 in its initial form; that’s a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements." The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive." Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI? 4. Even if AEGIS was an honest mix-up, that is itself a serious problem. Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.) Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for “operational excellence” in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run. In short The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive.

Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate. However, I feel like your response is still not really engaging with the heart of my concern about Anthropic's behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic. (Fair warning: this response is a bit long) To explain what I mean (and at the risk of being a bit esoteric), it’s worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropic’s role? Is that conception matched by how it actually behaves and impacts the world? One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion. Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders entitled “Core views on AI safety: When, why, what, and how.” In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in): “Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.” There have been more recent articulations of this version of Anthropic, for instance in Dario’s recent essay warning: “We must slow the pace at which we improve the capabilities of AI models… it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet… incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them… more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected…” A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropic’s AI systems can autonomously build their successors, and those successors build their successors, and so on. The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same. Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside. When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include: 1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at. 2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development. 3. Increased the percentage of AI development tasks that Claude “leads” from 0% in March of this year to 26% August (what is it in October?) 4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public. 5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance. 6. Was the first company to create an alternative weaker “frontier compliance framework” to supplement its RSP, which contains no binding provisions whatsoever on internal deployments. 7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were “best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.” 8. Commented “you can’t do safety from second place,” after the WH SI Accords. 9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems. Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization). But I think it’s worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that don’t reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadership’s explanations for this behavior at face value.) Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization). When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropic’s interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasn’t there a mistake pushing in the direction of being too strong? A core part of Anthropic’s theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice I’ve seen Anthropic push the frontier without substantially pushing the ‘safety frontier’ a remotely equivalent amount. I want to end by quoting from Anthropic co-founder Chris Olah’s comments around the Magnifica Humanitas release: “Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives. That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.” I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).

I'm glad OpenAI did comment ultimately on the choice to fire Jasmine, Tomek, and Mikita, though it should have been more substantive and happened earlier. The comms posture of being reactive on safety staffing issues, while being unclear with the policies about what information can and can't be shared, isn't going to solve the problem of restoring trust. I would very much like to see leaders from OpenAI prove that OpenAI is the best place in the world to do AI safety research and applied AI safety by making the information sharing policies extremely explicit. This set of issues isn't going away. This is the worst possible moment to alienate potential safety staff, given the importance of the issue on technical grounds and the salience of the issue to the public. I'd strongly recommend to OpenAI to endorse policy proposals that mandate disclosures from you and protect everyone who performs those disclosures through legally-established channels. Because the ambiguity is killing you, and you need to disambiguate fast. There's a risk of losing the best talent in the space and that just should not happen; we need all hands on deck at the frontier labs on safety going into ASI and RSI.

Reminder that Anthropic's frontier compliance framework does not have any provisions whatsoever about managing risks from internal models. Anthropic says that they "expect their approach to evolve" and "this framework will be updated" but its been months after HF and no change.

Some observations and reactions to this statement from OpenAI: • From the fired employees public communications, they seem to not understand why they were fired (the reasons they were given prior to their departure don't make sense). • OpenAI says in this note that "We will continue to be extremely forgiving of our team making good-faith mistakes" which implies that the employees in question were not acting in good faith. • I have previously commented that despite all my disagreements with OpenAI, I have genuinely admired their strong culture of staff speaking freely about their work and concerns. I am worried that the hazy circumstances around the firing of these employees risks jeopardizing that culture. • OpenAI seems to be saying that employees who are acting in good faith need not be worried about having their activities, including around safety work, chilled. But the fired employees are adamant that their actions were taken "in line with OpenAI's mission" and "within the working norms at the time." If OpenAI believes those norms need to shift, then they can make that decision (or if they dispute the employees claims that their actions were within the norms at the time, they can say so). But it seems clear that unless much more clarity is being provided internally, employees may reasonably respond by drawing back and ceasing to engage in open dialogue and collaboration that a month ago was considered normal and good. • To some degree the amount of interest that external folks like myself and the general public have in these firings is absurd. If United Airlines fired three members of their safety engineering team, it would not be that interesting or impact whether I felt confident boarding my flight, because I know that there are regulations and rules and norms that enable flight to be safe and reliable irrespective of the particular personnel. As @Miles_Brundage has said: "I don't think, before getting on a flight, about whether Bill McFlyerson's departure from Delta's leadership is a red flag or not. Nor should I have to care about this." But of course this is not the case for frontier AI development. • But in the current environment where all of these interactions are ad hoc and voluntary, I am worried that this episode may result in drawing back from some of the transparency and collaborations that can help make AI go well. Jasmine was one of the strongest voices internal to OpenAI advocating for pacing. Tomek was the main technical point of contact between outside evaluators and OpenAI during the Hugging Face incident investigation. Their departures in and of itself harm those sorts of initiatives, but if uncertainty and fear after their departure also results a broader drawback of permissible collaboration, then that would be even more harmful and concerning.

Worth reading. This sentiment is also how I feel some days working in AI policy in 2026 (though usually the answer I come to after some reflection is to continue working on what I was before, but somehow faster)

I strongly recommend reading the full letter from Mikita and his colleagues on their firing from OpenAI. They are raising vital issues that deserve everyones attention and the fact that they have been left guessing about the reasons for their simultaneous termination while working on essential safety work is not good. But also, between this letter and the recent White House Super Intelligence accords, I am realizing that in addition to the issues they raise there is a perhaps equally pressing need to improve the signatures of people who work on developing frontier models. Jasmine and Marks' are decent, but Dario, Greg, Mikita and Tomek should step it up if they plan on signing more important public documents in the future. (To be clear this is a joke and my handwriting is similarly unfit for public scrutiny)

More people should know about Psst - as they say here, they are representing the three fired OpenAI employees and offer a variety of pro bono services to current and former tech employees to help them know their rights and disclose important information to the public.

I don't know all the details, but Jasmine, Mikita and Tomek were all well respected core contributors to safety work at OpenAI, and it seems very concerning that all three of them do not seem to know why they were fired. The potential for a chilling effect is obvious.

“1. He left directory listing enabled on his own server, so researchers could figure out who he was with barely any effort. 2. They found files like CLAUDE.md and went through his Claude Code sessions. 3. There were questions like, “Where can I sell this data?” vibe code heist

One thing I think about a lot, ever since working at McKinsey, is that the single most robust finding in the industrial psych literature is that people's judgment and intelligence degrade when working sustained work weeks of over 40 hours, and essentially no one in a critical job ever takes this into account.

Great to see Senator Cantwell pushing for enforceable AI safety standards and independent testing, with explicit attention to loss of control and recursive self-improvement. Details will matter, but this framework looks serious and encouraging.

We at @LASST_law filed a lawsuit last week against OpenAI in response to its hack of Hugging Face in July. Some people asked why we have standing to bring this case in the first place. Here's an explanation. 🧵

Whether or not OpenAI deliberately held back attempting or publishing cryptographic breakthroughs, it seems clear that these internal models that are amazing at math will be very well suited to cryptographic breakthroughs and have massive ramifications for cybersecurity

Yesterdays deluge of novel math results from OpenAI are making me think of this quote from Lee Sedol (who was defeated by DeepMind's AlphaGo in 2017): "I faced the issues of A.I. early, but it will happen for others. It may not be a happy ending."

"If I were OpenAI, I would be asking this model to immediately target major unsolved problems in AI R&D." It seems safe to assume OAI and their peer companies are applying these internal models to AI R&D and this should inspire a feeling somewhat resembling severe vertigo