"Even “normal” general-purpose technologies are initially volatile and dangerous, even those we come to think of as unalloyed goods." Completely agree, and appreciate @DKThomp citing us making that very point. Emphasizing AI's similarity to past general purpose technologies should put more focus on this destabilization, not less. "Practically every major technological revolution has been, at least partly and briefly, an economic, social, or political catastrophe. From this reading of history, it would be odd if AI arrived without significantly destabilizing some part of our world for the worse."

An interesting cognitive bias: "GLM 5.3 is around as capable as Claude Mythos at cyber-exploitation while also having much lower safeguards, meaning that every additional day absent some dire cybersecurity incident is further evidence against catastrophic AI cyber-risk." That's right. And yet it doesn't seem to be treated that way by many people in the safety community. It's like they have a one-way valve for their risk estimates. I was curious if this bias has already been talked about in lesswrong / EA circles and indeed it has, going all the way back to this 2007 Yudkowsky post "Absence of Evidence Is Evidence of Absence"

New essay: A big-tent or small-tent AI safety movement? The unstated disagreement that underpins safety debates Two narratives about AI safety have emerged: either AI existential risk is real and imminent, or AI leaders’ and whistleblowers’ claims to that effect are insincere — a “psyop” or hype or a twisted form of regulatory capture. Few have considered the third possibility that x-risk warnings are sincere but simply wrong and counterproductive to AI safety. It’s as if everyone takes for granted that those raising the alarm are geniuses, and the only question is whether they’re the benevolent or the evil kind of genius. We subscribe to neither narrative. In our writing on AI safety, @sayashk and I consistently tried to take the arguments seriously but rebut the many claims we think are unsound while highlighting safety concerns that we think are real. This essay is a continuation of that agenda. Sections - It is true that we systematically underinvest in resilience against catastrophic and systemic risks - The Coxon incident shows how the warped media environment elevates x-risk over more grounded concerns - Why the x-risk framing may be counterproductive for AI safety policy - Conclusion: a big tent or small tent AI safety movement?

NeurIPS / Claude rejected our paper "Life After Benchmark Saturation" but we continue to think it is an important contribution to evaluation science, so we hope you check it out here!

I appreciate Anthropic’s transparency in sharing this chart but unsurprisingly it has led to speculative interpretations about intelligence explosion and superintelligence. I don’t think the chart implies we’re anywhere close to either. In short, task delegation ≠ task automation ≠ process automation ≠ faster progress ≠ recursive self-improvement ≠ intelligence explosion. Source: 1) The software engineering precedent: a year ago there were widespread hopes / fears that once AI can write ~100% of the code, software engineering output would explode (SaaSpocalypse! Everyone would create their own SaaS and dump their vendors) and that this would make software engineers obsolete. Since then, many companies and teams have basically hit that milestone, but neither of the assumptions proved true. Turns out we still need humans, and while shipping velocity has increased moderately, improvements in terms of actual outcomes for software users remain unclear. Besides, we’re still getting a better grasp on the negatives: code quality, long-term maintainability issues, and burnout. While the precedent is no guarantee, this should be our default expectation for what happens as the “Automation Level 4” line trends towards 100% — it won’t be a phase change. 2) The “production-progress paradox” is the fact that individual researchers’ productivity has been increasing while the rate of collective scientific progress has been slowing by most measures. AI exacerbates this because everyone uses the same or similar AI models, and ideas become homogenous over time. I suspect it’s too early to tell if this is going to bite companies that are plunging into AI-led research. Note: our own research on AI agents doing open-ended research shows limitations in creativity, judgment, and other areas. But it is possible that these could be overcome in the near future, so I’m discounting those limitations here. The production-progress paradox is a deeper issue that’s not AI-specific, though particularly applicable to AI-driven research. It’s about the fact that productivity increases are self-evident but true progress is not measurable as it happens (and only becomes clear in retrospect), so we end up optimizing for the wrong thing. 3) Let’s talk about automation level 5, which is still at 0% in Anthropic’s graph. It’s a bit unclear what Level 5 would look like, but it seems to be about full autonomy at the task level, and not the “AI builds its own successor” vision. My prediction for a while has been that even 100% task automation in most cognitive jobs won’t lead to any kind of discontinuity. What I expect will happen: if anything is understood well enough to be specifiable as a task, it can be handed off to AI, whereas the role of humans is entirely in the interstitial tasks — hard to formalize but still essential. So there will still be a human bottleneck. 4) This human bottleneck is a good thing and is essential for remaining in control. Humans don’t have to be in the loop on every task, but as long as there are enough touch points for oversight in the overall process, and adequate investment in improving human understanding and AI control, increasing AI capabilities doesn’t have to be bad for safety and, more broadly, collective human agency over AI. But “full RSI” is where this balance of agency can break. This kind of closed-loop process is arguably a much more important and tractable target for regulation than compute thresholds, superintelligence, or harm thresholds. I’m glad that OpenAI agrees that fully autonomous RSI may not be a good idea, in a just-released post: “Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely. Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices⁠ about the benefits and risks. Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand.” 5) Finally, many people have written about why Recursive Self Improvement, even if achieved, won’t necessarily lead to superintelligence. Here’s my argument: The bottlenecks are external.

OpenAI has talked a big game about AI for cyberdefense. But when @HacktronAI broke into their internal repository and reported it, they received a bug bounty of just $6,500 because one of the vectors for the attack was "out of scope". This is atrocious. If we actually want a flood of defenders auditing these systems, companies need to take bounties more seriously. Signing letters isn't enough.

Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators. These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad. We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes. Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground. To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should: 1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest 2. Rely on multiple evaluators with differing viewpoints and areas of expertise 3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings 4. Shield evaluators from retaliation 5. Grant access equivalent to that of highly privileged employees There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum. Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem. See the public letter here: Learn more at

I'm one of 100+ signatories to a public letter calling for five minimum requirements to make embedded evaluations credible, by establishing evaluator independence, protection from retaliation, and real access. ———— We, the undersigned, are encouraged to see frontier AI companies call for embedding third-party organizations to evaluate rapidly escalating AI capabilities and risks. We believe that all frontier AI companies should embed evaluators to independently assess AI risks, including evaluating the systems themselves and any significant incidents of real-world harm, as well as the companies’ training, deployment, oversight, operational, and safeguard practices. To be credible, embedded third-party evaluations must have scientific objectivity, transparency, independence, and robust protections against interference from the evaluated companies, including at least: 1. Frontier AI companies should rely on evaluators that are meaningfully independent, that maintain full editorial control, and that disclose and mitigate potential conflicts of interest. This includes at a minimum that embedded evaluation organizations should not be owned or governed by frontier AI companies, should not have other significant commercial business with them, and should not accept any form of payment or other reward contingent on the evaluator’s findings. 2. Frontier AI companies should incorporate differing viewpoints and areas of expertise, including by embedding multiple evaluation organizations across a range of priority risk areas, each with deep relevant technical expertise, as well as by allowing and encouraging evaluators to share how conclusions differ among evaluators and between evaluators and company employees. 3. Embedded evaluators should be transparent, including transparency about their methods and findings, the nature of their access, and the broader terms of the evaluation. Frontier AI companies should actively facilitate this transparency, including limiting the scope of non-disclosure agreements. They should also allow evaluators prompt and unfiltered communication with the companies’ boards and other privileged oversight bodies, as well as public release of findings and evidence, subject only to a time-limited redaction process restricted to protecting critical interests in intellectual property, customers’ sensitive information, individual privacy, security, and public safety. 4. Embedded evaluators should be shielded from retaliation from the companies they embed with for choosing reasonable evaluation methods, discovering information, or drawing conclusions that are unflattering to those companies. This includes reasonable protections against retaliatory litigation, as well as funding mechanisms that give them confidence they will remain funded even in these cases. 5. Frontier AI companies should grant embedded evaluators access equivalent to that of their own highly privileged employees for the purposes of their evaluations, and with exceptions to protect sensitive data belonging to the company’s customers and other third parties. This includes access to the same relevant systems, data, tools, and physical spaces as those available to senior internal company employees responsible for carrying out comparable risk assessments, as well as candid and direct one-on-one communication with relevant staff. This list is not comprehensive, and conditions like these to ensure credible evaluations should be increasingly standardized, codified, and enforced. One example is the set of terms defined in the AEF-1 standard, which has already seen early adoption, but far more work will be necessary to ensure that embedded evaluators are effective and meaningful. Embedded evaluations cannot address all oversight needs and should be treated as a complement to, rather than a replacement for, broader efforts by frontier AI companies to expand external oversight, including greater public transparency and additional, broader forms of access for independent researchers. ———— Full letter with links and signatures: I'm grateful to the AI Evaluator Forum (@aievalforum) for organizing this.

This is one of the central points in my ICML keynote, with a lot of detail — see part 2. Note: I take RSI seriously! One of our big empirical projects is evaluating agents' ability to do open-ended AI research. But it doesn't imply much about superintelligence, labor displacement, or doom.

Over 2 years ago @sayashk and I wrote a detailed deconstruction of p(doom) and argued that its primary function is to launder vague, evidence-free intuitions and fears through a facade of quantification. It remains 100% relevant today.

I think we have pretty good reason to accept that “AGI” is a meaningless term and a useless idea which should be retired. No one ever managed to agree on how you define AGI, but AI capabilities have improved enough to stress out Fields medalists, while the frontier is so “jagged” that regular users hate AI writing, and @random_walker was completely right about there being no discontinuity.

I've been using Pangram's Gmail inbox labeler for about two months now. It's been pretty huge to be able to see cold emails labeled as AI and use this info to inform how I want to engage. With this feature, there was a big discussion internally about privacy. Zero data retention (ZDR) -- where Pangram doesn't store the result after processing -- is a policy typically reserved for enterprise customers. However, since email data is much more sensitive, we ended up deciding to turn on ZDR for all email inbox scans. No email inbox data will be retained by Pangram, and this applies equally to everybody. The other really important thing here was to give users a choice of what happens to their email. My preference today is to simply label AI emails but leave them in my inbox. This may shift In the future if AI emails ramp up in volume, so we have the option to make fully AI emails skip the inbox or go straight to spam. Please try it out and let me know any feedback you may have!

Yes. Bio-risk exists today, from zoonotic outbreaks, accidents, and bio-terror. We know a great many things we can and should do to increase societal resilience, and have done almost none. Independent of how AI changes the risk, we should be doing the basics here. 1. Pathogen detection in waste streams, water supplies, and perhaps even indoor air. 2. Proactive design of template vaccines for every known virus family. 3. Pre-build vaccine manufacturing capability to have on standby. 4. Physical countermeasures: UV in the HVACs of all large buildings, airports, etc..; Stockpiling PPE. 5. R&D into new approaches like PCANS and other techniques that block viral transmission. These are all no-regret bio-defense policies that increase our resilience to natural pandemics, lab leaks, or intentional bio-attacks (AI augmented or not). Defense in depth.

Nihar Shah did a heroic experiment for TMLR: he spent 20-25 hours over two weeks interviewing authors of seemingly low-quality submissions about their own papers. He confirmed what we all suspected: people submitting these papers have *no idea* what is going on in them.

My toxic trait is loving even-handed, ecumenical takes. And @sayashk and @random_walker supply them in spades. A few thoughts and reactions: (1) This is what virtue looks like. Actual humility, a deep appreciation for uncertainty, openly updating their beliefs, and finding actionable areas of common ground. (2) Best line in the whole piece: “we do not need consensus on worldviews to have agreement on policy” — I’d put it even more strongly, such consensus is not possible; we have to work within that constraint. (3) Luckily many policies are robust to different assumptions. Their policy recs hold up, and that’s largely because they choose the right focus areas—Managing uncertainty and building resilience. I wrote more on the original policy recs in another post; I’ll link in the comments. (4) It’s striking how much their policy recs parallel the major AI policy proposals to date (including newer additions like embedded auditing and additional emphasis on liability reform). That still surprises many. (5) Despite these policies being chosen specifically for their robustness, most are hotly contested when actually proposed. I think the authors ought to ask why policies they’ve selected specifically for being agreeable no-brainers have not gotten traction amongst policymakers who like and cite their work. (6) I want to challenge the authors to consider whether they have unique leverage in unsticking some of these policies, and to act on it. (7) I do wonder if the AI as normal tech meme obscures the policy recs and encourages misreadings. I’d be curious for the authors to at least try writing some work that leads with the policy recs, and presents them in pithier form. The authors recognize this issue: “we are often mistaken as downplaying Al risks, though we have repeatedly clarified that that is not our position. Still, it is important for us to be explicit about how much urgency there is.” — but even now, I don’t think the urgency comes through to the average skim reader. (7) I’d like to see them incorporate ~adaptation/flexibility into their policy recs. Uncertainty = surprises and updates, and so many policies are too rigid to adapt. Maybe I’m just being a lawyer, but I’d like to see them stump for rulemaking authority, updating mechanisms for standards, etc. This is one of the most frequent errors in current policymaking and it seems very consistent with their thesis. (8) I think the authors, at times, underestimate how widely their takes are held amongst people they lump into the “AI Safety” category. I think a much larger portion of those folks agree with the diagnosis that the Hugging Face incident displayed large cultural and procedural safety lapses and an under-investment in control. There are legitimate disagreements, but I see the authors reaction as closer to the modal reaction than they seem to think. (9) I see this policy portfolio as highly overlapped with that of @law_ai_, and the emphasis on robustness to different assumptions has strong overlaps with both my own way of thinking and Radical Optionality

This strikes me as correct. I wonder what legal notice will/should look like when no one 1) opens mail; 2) answers their phone; 3) listens to voicemail; 4) reads texts from non-contacts; or 5) opens email. We're already in a version of this with spam but AI will make it worse

🙏🙏🙏The respect is mutual. A few thoughts on our relationship to AI safety: 1) We don’t see ourselves as adversarial to the community (the sentiment is not always reciprocated but that’s okay!). We care about AI safety, but we disagree on many of the specifics, and we think there’s value in epistemic diversity and in re-examining foundational assumptions and beliefs that the safety community tends to converge on too quickly. 2) We’ve emphasized at every opportunity, including in this essay, that there’s a lot of common ground on policy despite divergence in worldviews. We hope that this can help mitigate the polarization in policy debates that leads to chronic inaction. The fact that we have very different starting assumptions from the safety community is particularly useful in this regard. 3) We’ve spent a lot of time in conversation with the AI safety folks (Sayash is a regular at The Curve, for example) to share our perspective but also to learn and update our beliefs when warranted — something we do in this essay. Much of our empirical research is also about testing the cruxes of disagreement. 4) We don’t identify as part of the safety community but I hope it’s clear that there’s a lot of value in constructive dialog. We know the “normal technology” label irritates a lot of safety people — and I will write about the name at another point, explaining in more detail why we picked it and stand by it — but I think those who engage with our ideas and not just the title will find there’s a lot of common ground on safety!

There are things I disagree with here, but there is important stuff to learn from taking the perspective of parts of the cybersecurity industry that rogue AI incidents may be best understood as security & organizational failures that allowed rogue behavior to turn into problems.

An exceptional point here: Organizational competence is a hugely underrated piece of AI safety. There's a growing consensus at the frontier that we have to "pace," "go slow," or even pause. But there's no point in going slow just to go slow, or in pausing just to pause. If a company's RL still encourages misalignment, or if its sandboxes are poorly constructed, it doesn't matter that you're going 90mph or 60mph. Operational incompetency is unsafe at any speed. So, we need some way to smartly design, safely experiment with, share honestly, and maybe eventually mandate certain security protocols for advanced AI.