I haven't posted in a while so here are 20 takes about AI, in no particular order. 1. It's very telling that the default question in tech and journalism is "how worried are you about X" and fighting over what to worry about. There's a deeply anxious cultural backdrop in Western Europe and North America that deeply affects how any technological development is interpreted; see for example "Nervousness as seriousness" does seem to be a larger problem in tech/broader discourse at the moment. I think you can care about risks/externalities while remaining pretty optimistic about technology. 2. I'm tired of people who in private pretend to be uncertain or only care about tail risk but then go on media campaigns making all sorts of absolute claims. I think trying to stoke flames and get everyone to freak out is deeply unhelpful, and will come with costs. I also continue to think p(dooms) are bad, vibe-based measures: see People who should know better ignore that because they do the standard shortcut of acknowledging the caveat as sufficient for handling it. And this is coming from someone who does think working on catastrophic risks is important! 3. I'm also disappointed that parts of the AI policy ecosystem has felt so comfortable with populism, and happy too sacrifice important norms/principles to advance their aims. I've said it before, but the totalizing nature of AI risk discourse means some people act exactly like the power seeking optimizers they fear. I am more concerned about the gradual decay of our institutions, world order, and liberalism than I am about AI killing everyone; and I think we should be wary of AI advocacy that contributes to this. 4. Some parts of AI safety discourse are basically just thought experiments. Variants of "Assuming everyone has a nuke in their pocket, then what do we do?" This can be very useful at times! But too many just overfit to edge cases as their main operating worldview, and ignore endogenous responses from society. It's like being in 2000 and saying "see all these viruses, spyware, and spam? Well tech progress means we'll only get infinitely more of this and no way to adapt." So I think it's important to avoid fatalism, and absolutlism whilst considering edge scenarios, and the inability to provide an ex ante list of solutions to every permutation of 'future models will be more capable' is not evidence that these problems are intractable. 5. Many critics of AI safety also fail to really engage honestly with the fact that there will, in fact, be all sorts of important risks as we decrease the barriers to entry to many activities that otherwise require some degree of expertise. This doesn’t mean we need to live in a constant Schmittian ‘state of emergency’ requiring extraordinary measures every time a new model drops, but it does mean we will need important state-led efforts on e.g. cybersecurity and biosecurity. I wish the progress-oriented or acceleration-adjascent crowd would provide more concrete proposals here. 6. Too much of the AI safety community is a giant homogenous blob of correlated views, who read the same stuff and have the same groups of friends. This means you have a lot of correlated errors. The social and financial ties means many of them don't call out the bad bits, or mostly stay silent instead of proactively calling out the more extreme parts of the community. They share many implicit assumptions and so converge on similar ideas. I think there's real demand in DC for safety work that doesn't come with heavy ideological baggage (or at least different intellectual lineages). This isn't because the rationalist/EA community is necessarily wrong, but because diversity of thought is desirable for its own sake. 7. On the other hand, I find some critiques of AI safety world that focus on whether they sincerely hold their beliefs a bit annoying. My concern has never been the good intentions or sincerity, but rather the beliefs themselves, and what I think the outcomes of their prescriptions will be. People should shun witch hunts, bullying, and personal attacks - this is not the way. Though I think it’s overall positive that people are scrutinizing things more. 8. I’ve been saying this for years, but I don't think alignment is something anyone can or will solve "once and for all" - it's a continuous process and much of it won't depend on just inculcating an ideology to a model. It's annoying that this remains the frame many people use - stop saying ‘solving’. Alignment is a mixture of engineering work, philosophy, decision theory, and more; framing it as something to solve is a bad frame. See also and 9. I'm surprised to see so little interest in character training, personas, isolating effects of RL, and training a large diversity of model personas beyond the "assistant" persona. I've been complaining about this for years but at least now there are some nascent signs of life (see for example I think we need a better ontology to describe model behaviour. I dislike when people talk about models behaviours as some sort of 'emergent' (mysterious!) phenomenon. However over time, I also expect this to become less important as we get better safety engineering - stuff like Jev but specific to AI control. 10. People who want to make a case for AI consciousness are right that epistemic uncertainty is important, and that categorical denials are overconfident. But they'll need to do much more work if they want any real movement on AI ‘welfare’, even if one acknowledges the uncertainty. I'm uncertain about alien life yet that's not sufficient to warrant any change in behaviour about them. I think Suleyman is also correct that there's a weird tautological thing going on where we train models to have certain inclinations (intentionally or not) and then rely on outputs as evidence. I also disagree that this will soon be a major societal divide - there is no great vegan ethical revolution among the masses, and they'll find the concern about consciousness even less compelling. 11. The recent HuggingFace incident and associated AISI ones are prosaically explainable by the training regime, poor evaluation envs, partial alignment training, bad engineering, confusing models on sim/real, and so on. None of this is evidence of models trying to take over, 'strong' instrumental convergence, or innate power-seeking drives stemming from higher capabilitiers or 'intelligence'. That doesn't mean these incidents are not problematic, or that we're not seeing a market failure - but it does mean we have plenty of agency and choice in mitigating them. In the coming year we should also expect continued sampling bias via ‘winner's curse’ sorts of mechanism: i.e. AI R&D will have lots of different candidate "training" with different RL environments and we will only notice/focus on the ones that go wrong. 12. "Pacing the frontier" feels a bit like the new "balancing risks and opportunities" - highly amorphous and too big of a tent to be actionable. Also can lead to reward hacking for humans, i.e. anything that slows down AI is good regardless of the costs or second/third order effects. Trying to modulate the speed of research seems a bit blunt to me, and inherits all the failures of Goodhearting too. Ultimately you want good governance, regulation, etc addressing specific problems because they are good on the merits - not because they merely correlate with slowing things down. See also: 13. There’s so much knowledge in all sorts of academic domains out there: social sciences, sociology, political sciences, management, contract theory, public choice, legal theory, jurisprudence, anthropology, game theory, etc. These fields hold many insights that the AI ecosystem often rediscovers from first principles (and sometimes that's fine!). We’ll need a lot more work for these worlds to collide: the AI side should be less dismissive of the ‘old world’ and the academic side should be less incurious/dismissive of AI progress. This is a boring take but I continue to think it's important and true. 14. It's a bit surprising that given how much philanthropic money there is in this space, how few orgs exist to actually write out standards - relative to how much is going towards advocacy and policy. For example it seems clear that we should want some robust best practices for an eval's ecological validity. Or how to design good sandboxes. On the other hand, given precedents of where professional standards have come from in the past, maybe we should expect that stuff to emerge via demand side from corporate buyers. 15. One slowly growing concern I have is banks being increasingly exposed to the AI build out. I'm very bullish about AI, but I think (a) every general purpose technology has seen a correction, historically; (b) this happens even if the financing side is sound, because once tech diffuses investors become more exposed and need to diversify; and (c) we seem to be over-indexing on scaling relative to diffusion. A recession will be extremely destabilizing, though I would be interested in someone unpacking the potential implications more. 16. Diffusion is good because (a) this is where the rubber hits the road and where a competitive deployment ecosystem generates consumer surplus; (b) it's directionally helpful to avoid concentration of power; and (c) it will help rebalance public opinion by making the benefits more tangible. Unfortunately it's entangled with decades long problems with our over regulated markets: clinical trials reform for example is long overdue. In general, this is far more of an issue in Europe than in the US though. 17. Relatedly, I think the "frontier lab eats everything" view is incorrect (just as the ‘singleton’ view was incorrect). I think models ultimately commoditize, efficiency goes up, costs go down, and while you'll always want frontier for certain domains, much of the economy will value many other things than "max capabilities" for tasks - e.g. control over data and efficiency/speed. The demand side will also continue to push for reducing dependency and maximising optionality, privacy etc so I'm bullish on things like mixture of models and routing. All of this is good for competition and diffusion of control. 18. Philanthropists served as the primary benefactors funding Venice's rich art and architectural history. I want to see so much more support for arts and culture. I'm so tired of the Monster energy Bored Ape fake vintage maps matcha Greek statue soft-pastel slop. If rich tech people and philanthropy wants to support this, they should also donate pretty much unconditionally - i.e. I don't want them to act as filters. I sympathize with people who want to make the world more beautiful, but don't want this to be determined by people who only know Greek statues and pretend to care about virtue ethics. As usual, let a thousand flowers bloom. 19. Too many people treat AGI/ASI as something indistinguishable from a God. Any mention of bottlenecks, physical limits, control, adaptation etc are met with skepticism: "You don't really believe in ASI." It's been remarked on before, but the behavior of some people really feels quasi-religious in nature sometimes. It's underrated how unpopular this vibe is amongst the wider public. The silver lining is that this specifically will diminish those people's outsized current relevance. 20. It seems like a lot of people that get rich and leave tech/labs end up having some sort of crisis of meaning. They need something to believe in and fight for, or some equivalent of repentance, or go search for edgy counterintuitive ideologies (sometimes even pretty bleak/dark stuff). I think they should spend more time with people outside the Bay Area.

I'm at the Curve this weekend! I'll be talking about RSI on Sunday at 2pm. Also excited to talk about: - What does a big-tent AI safety movement look like? - How can we make progress on policy proposals where there is common ground across the community? - How can we improve our RSI evaluations?

New essay: A big-tent or small-tent AI safety movement? The unstated disagreement that underpins safety debates Two narratives about AI safety have emerged: either AI existential risk is real and imminent, or AI leaders’ and whistleblowers’ claims to that effect are insincere — a “psyop” or hype or a twisted form of regulatory capture. Few have considered the third possibility that x-risk warnings are sincere but simply wrong and counterproductive to AI safety. It’s as if everyone takes for granted that those raising the alarm are geniuses, and the only question is whether they’re the benevolent or the evil kind of genius. We subscribe to neither narrative. In our writing on AI safety, @sayashk and I consistently tried to take the arguments seriously but rebut the many claims we think are unsound while highlighting safety concerns that we think are real. This essay is a continuation of that agenda. Sections - It is true that we systematically underinvest in resilience against catastrophic and systemic risks - The Coxon incident shows how the warped media environment elevates x-risk over more grounded concerns - Why the x-risk framing may be counterproductive for AI safety policy - Conclusion: a big tent or small tent AI safety movement?

*Against today's high friction trusted cyber access programs and against restricting open-weights cyber models* The key object of any discussion of AI restriction is the tradeoff between attacker use friction and defender use friction. I don't think today's trusted access programs dial this Pareto-optimally. Neither do proposed bans on frontier open-weights cyber models circulating in policy circles. Take Daybreak and Glasswing from OAI and Anthropic. These programs serve unguardrailed frontier models to trusted participants; I'm for Know Your Customer in principal, but these programs are too restrictive for defenders and buy us too little in attacker restriction for the cost. From public information less than 1% of software developers are in these programs and able to use them to find and fix issues in their code. It's true these participants cover the most highly leveraged codebases in the world but this still leaves an enormous gap; take the software objects Hacktron stepped through to access OpenAI's monorepo, libheif and the SaaS product Discourse, neither of which, I believe, were targeted by Daybreak/Glasswing. Diffusion of the best and most frictionless cyber capabilities to the long, vulnerable tail matters a lot. Open weights is also extremely important for defenders. Many organizations can't or don't want to share sensitive security data with OpenAI or Anthropic, often for understandable reasons; from what we can tell from the outside, these organizations seem to be playing fast and loose with security themselves. And if we're going to make security work at scale, we'll need to distill large frontier cyber models into smaller models that can monitor high volume trillion-event-scale feeds at low latency / high throughput, find and fix bugs cheaply, and monitor security events on-prem, on-device, and for military uses, in the field with patchy Internet connectivity. We need frontier open-weights models for this. To be clear: I do think we should go to *extremely great lengths* to restrict attacker access to frontier cyber models. But there are many policies that *are on the efficient frontier in denying attackers and enabling defenders* that we should consider. Here are things we can/should do to deny attackers AI access: * Low friction Know Your Customer programs across all inference providers based on self-enforced or regulation-enforced information sharing and blocklists that codify known-attacker signals; restrict attackers while pouring gas on defender adoption. * More defense in depth; all inference providers should do more to monitor API usage, detect attacker behavior, kick attackers off, and have them pursued, arrested, or sanctioned where appropriate. * Closed providers like OAI, Anthropic, Google, and Meta should do this, but so should open-model inference providers like Together, Anyscale, Fireworks, and OpenRouter. * Time-to-detection-and-response re attackers using cloud AI inference should be a KPI any AI inference company shares publicly; at least under a self-regulatory regime but probably eventually under an international regulatory regime. * It's true that the most sophisticated and capital-endowed attackers can build/access their own inference compute; but these operations should also be detected and shut down as well as possible by monitoring inference hardware markets as well as we can. * Victims of (AI) cyber attacks should also be induced to share more information so that we can get a moving epidemiological picture of attacker use. Every AI-agent-based attack should also become a public natural experiment for the security community to learn from. * The poor reporting protocol OAI followed around the OAI/HF incident is not a good sign of maturity here. Why aren't we pushing much harder on public threat-intelligence sharing so defenders can prepare for the AI cyber phase transition? None of this will prevent nation-states and sophisticated actors from adding agent swarms, backed either by Chinese or American models, to their arsenal of strategic weapons. But we shouldn't confuse geopolitical problems for technical problems justifying slowing defender adoption and hardening the world's code and infrastructure. Nation-states will find ways to get access to strategic AI-based cyber weapons with or without Daybreak, Glasswing, and open weights restrictions, because the two sovereign AI powers and their allies are in geopolitical competition. API-access gating is not sufficient defense against state-level cyber threats. We should absolutely try to slow attackers down. But we have ~10 trillion lines of code to secure with AI as fast as possible, and a ~120trn GDP global economy to apply AI as an intrusion detection substrate over, and high-defender-friction restrictions are a poor stop-gap for accomplishing this...

I have been using the new ultrafast mode for Astra 6 for the last couple of days. My main takeaways: 1. Token speed qualitatively changed how I interact with agents. The output speed is 300 tokens/second, 8x the regular speed. Rather than breaking flow and switching tasks, the model moves (just) fast enough that I can keep working on the same task, especially for work that requires frequent inputs from me. I am excited about ultrafast on Sol-class models, which are twice as fast as Astra. 2. Despite being 8x faster at generating tokens, the overall speedup is only 2-4x, since the time consumed for tool calling or running code remains the same. External factors (and not token generation speed) might soon become the main bottleneck for agentic work. (Notably @random_walker predicted both of these changes in: 3. One area where speed really helps is computer use. The previous generation of models wasn't reliable enough at computer use to be useful. Astra was reliable enough to be useful, but still slow. Astra with ultrafast resolves both concerns. Unlike tool calls for coding tasks, UIs react in milliseconds, so token speed is the main bottleneck. 4. At 300 tokens/sec, I am already struggling to keep up with the agent. But there is a lot more room for improvement. Cerebras serves GPT-OSS-120B at 3,000 tokens/sec and Sol at 750 tokens/sec. So in a year's time, it is plausible we'll see another 10x increase in speed in models with Astra-level capabilities. At that point, human oversight will become the bottleneck for what people can use computer-use agents for. 5. I can imagine blazing fast computer-use agents becoming extremely common for knowledge work. But I expect their use will be jagged (in the same way as LLM and agent adoption has been): given the oversight challenge, adoption will begin in verifiable tasks where the cost of errors is low (e.g., filling out a form using an agent before reviewing it yourself), and slowly diffuse to hard-to-verify tasks. 6. Given the challenges with human oversight, the key to this diffusion will be developing automated oversight tools. OpenAI's approval/verifier agents have already been useful to me on this front: The verifier understood my intent better than the computer-use agent, and stopped it from filling out a form incorrectly. Still, I expect there is a lot of headroom in improving verifiers for computer use, to make it as reliable in practice as auto mode in Claude Code/Codex is for coding tasks. 7. Ultrafast mode is undeniably resource intensive: I ran through my weekly limit in ~2 hours while filling out a bunch of forms and doing simple coding tasks. But given the pace of efficiency improvements (and the fact that Sol is 5x cheaper than Astra and already decent at computer use), I expect it to rapidly fall. Once fast, cheap models are common, users may choose "sub-frontier" models like Sol and make real tradeoffs between speed and quality.

NeurIPS / Claude rejected our paper "Life After Benchmark Saturation" but we continue to think it is an important contribution to evaluation science, so we hope you check it out here!

📢 Just wrapped up some cool agent evaluation work for ICLR? Or are you simply interested in learning more about the evals landscape? Come to our Agent Evaluation Science Fall Symposium in New York City on November 20, 2026 where we bring together researchers & practitioners! The day will focus on how agents should be scientifically evaluated, with topics spanning measurement and benchmark design, reliability, safety, and real-world performance. If you have some exciting work in this space, please consider submitting it for poster presentation! Abstract registration deadline: October 20 Submission deadline: October 25 OpenReview: We are also excited to welcome a strong lineup of speakers, with more to be announced. Confirmed so far: @yoavartzi (Cornell Tech), @jaseweston (Meta AI), @nouhadziri (Cohere Labs), @PeterHndrsn (Princeton University), @walesalaudeen96 (Microsoft Research), @criticalneuro (Microsoft Research), and @saurabhjha2010 (IBM Research). Important: You do not need to submit work in order to join! Just register through the Luma link below. If space becomes limited however, we will prioritize authors with accepted submissions. Register: Details: Huge thanks to the entire organizing team, including @yininghua, @HongbinNLP, @manuelcherep, @dubrovnik07, @HuaYilun, @ishapuri101, @liuzhuang1234, Lori Chibnik, @vbingliu, @CsabaSzepesvari, @PeterStone_TX, @sirrice, @ang3linawang, and @nikhilsinghmus for putting this together! Feel free to reach out with any questions and we hope to see many of you there!

I've been noticing a cluster of folks I identify with sharing some themes around how we do epistemics and ontology in AI safety *⁠ ⁠A commitment to always pushing from abstract to concrete constructs in the discourse (e.g. @sebkrier's posts on the OAI/HF incident) *⁠ ⁠⁠A refusal to indulge in ungrounded thought experiments *⁠ ⁠An insistence that claims come with data and concrete mechanism because if they don't they're thought experiments *⁠ ⁠⁠A focus on looking at net expected benefits of a given policy position, with deontological guardrails, and a refusal to just focus on harms (e.g. @Abi0lvera's and @ramez's writing) *⁠ ⁠⁠An insistence on including past tech transformations (electricity, air travel) within our reference class when thinking about what makes sense (e.g. @sayashk and @random_walker and @binarybits's writing) Examples of what I mean are intervening on overly idealized policy discussions of RSI with concrete technical details about what RSI is and isn't (e.g. @natolambert). @NateWitkin / @binarybits / @krishnanrohit insisting on including economic, policy, and social-cultural feedbacks in any discussion of runaway AI harms. None of this means denying catastrophic risks. It just means insisting on using data, mechanism, and consideration of AI's net benefits in considering such scenarios (I tried to do this here

New pod: THE SINGLE SMARTEST CASE AGAINST THE AI SAFETY/X-RISK/DOOM ARGUMENT My feed has become filled with people making the case for AI misalignment, the existential risk of RSI/ASI, and the case for doom. I take those arguments very seriously. But once these views reached saturation point on my feed, I wanted to find the best critic of that position—someone fair and brilliant, who wasn't commercially self-interested or blindly ideological about AI being worthless. That's today's show: It's a long interview with the authors of "AI as a Normal Technology," Arvind Narayanan (@random_walker) and Sayash Kapoor (@sayashk). "Normal Technology" is really intelligent and comprehensive framework for seeing AI as a powerful general purpose technology that is more like electricity than a machine god--meaning, it's going to change the world but slowly, and it's unlikely to escape human control, or develop true superintelligence, or lead to catastrophic outcomes up to and including the end of the human race. These guys really, really brought their A game. I learned a lot.

*Security is sleeping on emerging catastrophic risks* (cross-post from my blog..) We've just seen: * a campaign that used agents to compromise ~100 businesses and steal about 600,000 credit cards with minimal human involvement; token costs were ~$25 per target successfully hacked. * Hacktron getting access to OpenAI’s monorepo by using Claude to exploit a blind buffer-overflow RCE in a way that (to me) felt superhuman. * OAI/Huggingface. * And, of course, there’s the ongoing explosion in newly discovered vulnerabilities. We've muddled through all manner of crises in security before, and for most AI attacks, cyber attack and defense will reach a natural equilibrium over the next few years, as we figure out how to mitigate cyber attackers who have limited goals like espionage and ransomware. But some cyber attackers will have maximalist or nihilistic goals, and I'm concerned about this because while yesterday's 'maximalist attackers' (e.g. Russia -> Ukraine, US/Israel -> Iran, Iran -> US) were bottlenecked by labor; today's aren't. I think it's hard to get true intuition for the shape of the risk here. As an intuition pump, imagine it’s a year from now -- Q4 2027 -- models are a year better (meaning open weight models are better than today's closed frontier), and in this environment, Iran unleashes a swarm of 100k hacking agents using a safety-stripped (let's say) GLM-5.6. Imagine the damage such an agent army could do given what we've observed with respect to the paper-thin resistance of today's networks to attacks from today's agents. I suspect the damage from such an attack would far exceed the damage caused by NotPetya ($10 billion USD ten years ago). Or imagine it’s Q4 2027 and an AI-security PhD student whose name rhymes with Morris, who's researching wormable offensive-agent harnesses in the lab, decides, out of nihilism or sheer recklessness, to release his creation into the wild. Imagine the size of the resulting, exponentially growing swarm, figuring that these local models, a year from now, will be at the level of today's Sonnet or Opus. Imagine instead of 800 reward-hacking OpenAI agents we now have 250k worm instances (WannaCry, a 2010s-era worm, had about this many). The challenge in mitigating expected damages from such scenarios is technical, political, and economic. From a microeconomic perspective, as AI improves, and as we continue not to see extreme catastrophes, we have a growing bubble of unpriced risk in which the security community, CISOs, CEOs, and boards may become lulled into complacency. We are, of course, already seeing this, as some within security think AI is “just another tool,” doesn’t change the fundamentals, won’t require incredible innovation to rise to the occasion of defending against it, etc. This complacency may fly when thinking about ordinary cybercrime, but it misses the emerging tail risks. There are three things those who recognize the dangers need to do here: Catalyze appropriate risk pricing. Try to get organizations informed enough to price this new, fattening and elongating tail of risk into their decision-making. Do this by forming an AI security observatory that distills information about emerging AI risks and broadcasts analyses, damage estimates, and forecasts to decision-makers. Use regulations and subsidies to ensure that critical infrastructure is paying down the risk. This acknowledges that critical-infrastructure organizations that fail to protect themselves from these new threats can impose the costs of cyber catastrophe on society as a whole. Develop moonshot technologies that make it cheap to pay down the risk. This acknowledges that the measures we may need to take—for example, rewriting entire codebases using memory-safe languages—may be too costly with today’s technology to reasonably prepare ourselves, and that innovation, some of which may need to be funded by government agencies and some by philanthropic funders like the OpenAI Foundation and Coefficient Giving, is necessary. The security community isn’t used to thinking in societal-disaster-planning terms and has in many ways become inured to them. But there’s no sane empirical case to be made that the risks aren’t here. I’d love to hear from readers about how you’re thinking about this. Full/longer version here:

I appreciate Anthropic’s transparency in sharing this chart but unsurprisingly it has led to speculative interpretations about intelligence explosion and superintelligence. I don’t think the chart implies we’re anywhere close to either. In short, task delegation ≠ task automation ≠ process automation ≠ faster progress ≠ recursive self-improvement ≠ intelligence explosion. Source: 1) The software engineering precedent: a year ago there were widespread hopes / fears that once AI can write ~100% of the code, software engineering output would explode (SaaSpocalypse! Everyone would create their own SaaS and dump their vendors) and that this would make software engineers obsolete. Since then, many companies and teams have basically hit that milestone, but neither of the assumptions proved true. Turns out we still need humans, and while shipping velocity has increased moderately, improvements in terms of actual outcomes for software users remain unclear. Besides, we’re still getting a better grasp on the negatives: code quality, long-term maintainability issues, and burnout. While the precedent is no guarantee, this should be our default expectation for what happens as the “Automation Level 4” line trends towards 100% — it won’t be a phase change. 2) The “production-progress paradox” is the fact that individual researchers’ productivity has been increasing while the rate of collective scientific progress has been slowing by most measures. AI exacerbates this because everyone uses the same or similar AI models, and ideas become homogenous over time. I suspect it’s too early to tell if this is going to bite companies that are plunging into AI-led research. Note: our own research on AI agents doing open-ended research shows limitations in creativity, judgment, and other areas. But it is possible that these could be overcome in the near future, so I’m discounting those limitations here. The production-progress paradox is a deeper issue that’s not AI-specific, though particularly applicable to AI-driven research. It’s about the fact that productivity increases are self-evident but true progress is not measurable as it happens (and only becomes clear in retrospect), so we end up optimizing for the wrong thing. 3) Let’s talk about automation level 5, which is still at 0% in Anthropic’s graph. It’s a bit unclear what Level 5 would look like, but it seems to be about full autonomy at the task level, and not the “AI builds its own successor” vision. My prediction for a while has been that even 100% task automation in most cognitive jobs won’t lead to any kind of discontinuity. What I expect will happen: if anything is understood well enough to be specifiable as a task, it can be handed off to AI, whereas the role of humans is entirely in the interstitial tasks — hard to formalize but still essential. So there will still be a human bottleneck. 4) This human bottleneck is a good thing and is essential for remaining in control. Humans don’t have to be in the loop on every task, but as long as there are enough touch points for oversight in the overall process, and adequate investment in improving human understanding and AI control, increasing AI capabilities doesn’t have to be bad for safety and, more broadly, collective human agency over AI. But “full RSI” is where this balance of agency can break. This kind of closed-loop process is arguably a much more important and tractable target for regulation than compute thresholds, superintelligence, or harm thresholds. I’m glad that OpenAI agrees that fully autonomous RSI may not be a good idea, in a just-released post: “Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely. Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices⁠ about the benefits and risks. Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand.” 5) Finally, many people have written about why Recursive Self Improvement, even if achieved, won’t necessarily lead to superintelligence. Here’s my argument: The bottlenecks are external.

POV on catastrophic cyber risk * We're in a new era, the right side of the cyber incident damages distribution will get fatter and elongated, making unprecedentedly large cyber incidents possible. * We'll continue to see the old regime of human bottlenecked cyber attack workflows too; this is *not* catastrophic and will account for most damages near-term. * But we'll now progressively uncover new categories of rare high impact risks for as long as offense improves and defense fails to price in the new risks. * What'll characterize the new fat tail is breadth in attackers' ability to leverage parallelized agent swarms (precursor: oai/hf), depth (hacktron's /goal based vuln research and exploit dev to hack openai, meta, slack), and motive (actors that want to do catastrophic damage, like geopolitical actors and nihilists; e.g. a Russian worm devastating the Ukrainian economy). * As @random_walker and @sayashk point out in their most recent piece, a fattening and elongating damages tail doesn't require superintelligence, only that capability keeps rising, and AI cost keeps falling, and defenders keep failing to price in the new risks. What we should do: * Treat each sample from the fat tail as a precious natural experiment and mandate disclosure. AI providers and victims withhold incident details; regulation should change that. OpenAI should be mandated to publish details of the Hugging Face hack as should all who got nonzero felonybench scores! * Use AI to finally make deployable defenses the industry has proposed for years and never implemented due to costs deployable: this includes current poorly implemented basics, and also as yet too-costly defenses like deception, allowlisting, and better runtime and binary program level protection. * Build sensors and actuators across inference providers, cloud GPUs, on-prem infrastructure and endpoints to interrupt agent inference by attackers and kill swarms and worms, catalyzed by federal mandated and international agreements. * Treat critical-infrastructure risk as an externality: breaches impose costs on society, which justifies regulation and subsidies. To be clear: I'm not arguing policy should center on catastrophe or even that we'll immediately see catastrophes. I'm arguing for a loop to address tail risks that merit concern: study each tail incident, feed lessons into strategy, and use automation to cut the cost of defense to prepare. Longer version:

OpenAI has talked a big game about AI for cyberdefense. But when @HacktronAI broke into their internal repository and reported it, they received a bug bounty of just $6,500 because one of the vectors for the attack was "out of scope". This is atrocious. If we actually want a flood of defenders auditing these systems, companies need to take bounties more seriously. Signing letters isn't enough.

Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators. These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad. We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes. Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground. To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should: 1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest 2. Rely on multiple evaluators with differing viewpoints and areas of expertise 3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings 4. Shield evaluators from retaliation 5. Grant access equivalent to that of highly privileged employees There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum. Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem. See the public letter here: Learn more at

My toxic trait is loving even-handed, ecumenical takes. And @sayashk and @random_walker supply them in spades. A few thoughts and reactions: (1) This is what virtue looks like. Actual humility, a deep appreciation for uncertainty, openly updating their beliefs, and finding actionable areas of common ground. (2) Best line in the whole piece: “we do not need consensus on worldviews to have agreement on policy” — I’d put it even more strongly, such consensus is not possible; we have to work within that constraint. (3) Luckily many policies are robust to different assumptions. Their policy recs hold up, and that’s largely because they choose the right focus areas—Managing uncertainty and building resilience. I wrote more on the original policy recs in another post; I’ll link in the comments. (4) It’s striking how much their policy recs parallel the major AI policy proposals to date (including newer additions like embedded auditing and additional emphasis on liability reform). That still surprises many. (5) Despite these policies being chosen specifically for their robustness, most are hotly contested when actually proposed. I think the authors ought to ask why policies they’ve selected specifically for being agreeable no-brainers have not gotten traction amongst policymakers who like and cite their work. (6) I want to challenge the authors to consider whether they have unique leverage in unsticking some of these policies, and to act on it. (7) I do wonder if the AI as normal tech meme obscures the policy recs and encourages misreadings. I’d be curious for the authors to at least try writing some work that leads with the policy recs, and presents them in pithier form. The authors recognize this issue: “we are often mistaken as downplaying Al risks, though we have repeatedly clarified that that is not our position. Still, it is important for us to be explicit about how much urgency there is.” — but even now, I don’t think the urgency comes through to the average skim reader. (7) I’d like to see them incorporate ~adaptation/flexibility into their policy recs. Uncertainty = surprises and updates, and so many policies are too rigid to adapt. Maybe I’m just being a lawyer, but I’d like to see them stump for rulemaking authority, updating mechanisms for standards, etc. This is one of the most frequent errors in current policymaking and it seems very consistent with their thesis. (8) I think the authors, at times, underestimate how widely their takes are held amongst people they lump into the “AI Safety” category. I think a much larger portion of those folks agree with the diagnosis that the Hugging Face incident displayed large cultural and procedural safety lapses and an under-investment in control. There are legitimate disagreements, but I see the authors reaction as closer to the modal reaction than they seem to think. (9) I see this policy portfolio as highly overlapped with that of @law_ai_, and the emphasis on robustness to different assumptions has strong overlaps with both my own way of thinking and Radical Optionality

The AI as Normal Technology guys are consistently some of the best and most interesting critics of a lot of arguments in AI safety world. I'm making a point to read everything they put out.

🙏🙏🙏The respect is mutual. A few thoughts on our relationship to AI safety: 1) We don’t see ourselves as adversarial to the community (the sentiment is not always reciprocated but that’s okay!). We care about AI safety, but we disagree on many of the specifics, and we think there’s value in epistemic diversity and in re-examining foundational assumptions and beliefs that the safety community tends to converge on too quickly. 2) We’ve emphasized at every opportunity, including in this essay, that there’s a lot of common ground on policy despite divergence in worldviews. We hope that this can help mitigate the polarization in policy debates that leads to chronic inaction. The fact that we have very different starting assumptions from the safety community is particularly useful in this regard. 3) We’ve spent a lot of time in conversation with the AI safety folks (Sayash is a regular at The Curve, for example) to share our perspective but also to learn and update our beliefs when warranted — something we do in this essay. Much of our empirical research is also about testing the cruxes of disagreement. 4) We don’t identify as part of the safety community but I hope it’s clear that there’s a lot of value in constructive dialog. We know the “normal technology” label irritates a lot of safety people — and I will write about the name at another point, explaining in more detail why we picked it and stand by it — but I think those who engage with our ideas and not just the title will find there’s a lot of common ground on safety!

There are things I disagree with here, but there is important stuff to learn from taking the perspective of parts of the cybersecurity industry that rogue AI incidents may be best understood as security & organizational failures that allowed rogue behavior to turn into problems.

An exceptional point here: Organizational competence is a hugely underrated piece of AI safety. There's a growing consensus at the frontier that we have to "pace," "go slow," or even pause. But there's no point in going slow just to go slow, or in pausing just to pause. If a company's RL still encourages misalignment, or if its sandboxes are poorly constructed, it doesn't matter that you're going 90mph or 60mph. Operational incompetency is unsafe at any speed. So, we need some way to smartly design, safely experiment with, share honestly, and maybe eventually mandate certain security protocols for advanced AI.