Register and share your invite link to earn from video plays and referrals.

Bronson Schoen
@BronsonSchoen
MTS @ Apollo Research, prev: Nvidia, Apple
2.4K Following    1.1K Followers
GPT-6 Astra is just insanely good at detecting control interventions in different environments (multi-step essay writing, single step coding and agentic settings). But other models such as Fable 5 and Opus 5 are also much better than older models.
Show more
New in The Atlantic: @dgrobinson resigned this week. He was among the longest-tenured employees at OpenAI—and oversaw safety reports on 12 frontier launches. He is very worried: “The time for trial and error is over.” You can read his essay here:
Show more
0
76
1.3K
365
Forward to community
It seems really underappreciated to me for various threat models how much organizational and live information the models seem to have access to from connections like slack, gdocs, or whatever equivalents a lab uses. Like can the models just watch slack for if they’re caught? Do they know about gaps in monitoring / results from collusion evals? Can they send messages and use slack as shared state? Etc.
Show more
New OpenAI misalignment disclosures! 1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack. We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.
Show more
To develop frontier AI safely, developers must be able to show that their models are not scheming, i.e. covertly working against them in pursuit of unintended goals. New post: four claims any scheming safety case must make, and the resources embedded evaluators need to verify them 🧵
Show more
While AI chip exports get more attention, whether China can continue to import ASML’s DUV immersion (DUVi) lithography machines will be the decisive factor in US-China AI competition over the next decade. @techstatecraft has published a 30,000-word report that models how continued DUVi exports may let Huawei and other Chinese chipmakers scale AI chip production to compete with leading US AI chipmakers like NVIDIA. Our core findings: (1) 𝗖𝗵𝗶𝗻𝗮 𝗻𝗲𝗲𝗱𝘀 𝗗𝗨𝗩𝗶 𝗺𝗮𝗰𝗵𝗶𝗻𝗲𝘀 𝘁𝗼 𝗺𝗮𝗸𝗲 𝗔𝗜 𝗰𝗵𝗶𝗽𝘀, 𝗯𝘂𝘁 𝗶𝘀 𝘂𝗻𝗹𝗶𝗸𝗲𝗹𝘆 𝘁𝗼 𝗰𝗼𝗺𝗺𝗲𝗿𝗰𝗶𝗮𝗹𝗶𝘇𝗲 𝗶𝘁𝘀 𝗼𝘄𝗻 𝗮𝘁 𝘀𝗰𝗮𝗹𝗲 𝘂𝗻𝘁𝗶𝗹 𝘁𝗵𝗲 𝗺𝗶𝗱-2030𝘀. This gives the United States and its allies, the sole producers, extraordinary leverage over China’s AI ambitions. (2) 𝗬𝗲𝘁 𝘁𝗼𝗱𝗮𝘆 𝗔𝗺𝗲𝗿𝗶𝗰𝗮 𝗮𝗻𝗱 𝗶𝘁𝘀 𝗮𝗹𝗹𝗶𝗲𝘀 𝗮𝗿𝗲 𝗴𝗶𝘃𝗶𝗻𝗴 𝗮𝘄𝗮𝘆 𝘁𝗵𝗮𝘁 𝗹𝗲𝘃𝗲𝗿𝗮𝗴𝗲 𝗯𝘆 𝗽𝗲𝗿𝗺𝗶𝘁𝘁𝗶𝗻𝗴 𝗖𝗵𝗶𝗻𝗮 𝘁𝗼 𝗶𝗺𝗽𝗼𝗿𝘁 𝗗𝗨𝗩𝗶 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 𝗰𝗮𝗽𝗮𝗯𝗹𝗲 𝗼𝗳 𝗺𝗮𝗸𝗶𝗻𝗴 𝗛𝘂𝗮𝘄𝗲𝗶'𝘀 𝗔𝗜 𝗰𝗵𝗶𝗽𝘀. China has aggressively capitalized on this opening, amassing a large, growing stockpile of these systems as of 2026. If Chinese DUVi imports continue, China could accumulate enough lithography capacity over the next five to ten years to narrow or even erase the currently sizable US-allied advantage in AI chipmaking. (3) 𝗧𝗵𝗲 𝗨𝗻𝗶𝘁𝗲𝗱 𝗦𝘁𝗮𝘁𝗲𝘀 𝗰𝗮𝗻 𝗽𝗿𝗲𝘃𝗲𝗻𝘁 𝘁𝗵𝗶𝘀 𝗼𝘂𝘁𝗰𝗼𝗺𝗲 𝘄𝗶𝘁𝗵 𝗮 𝘀𝗶𝗺𝗽𝗹𝗲 𝗽𝗼𝗹𝗶𝗰𝘆 𝗰𝗵𝗮𝗻𝗴𝗲: 𝗿𝗲𝗽𝗹𝗮𝗰𝗲 𝘁𝗵𝗲 𝗰𝘂𝗿𝗿𝗲𝗻𝘁, 𝗽𝗮𝗿𝘁𝗶𝗮𝗹 𝗗𝗨𝗩𝗶 𝗰𝗼𝗻𝘁𝗿𝗼𝗹𝘀 𝘄𝗶𝘁𝗵 𝗮 𝗖𝗵𝗶𝗻𝗮-𝘄𝗶𝗱𝗲 𝗯𝗮𝗻 𝗼𝗻 𝗮𝗹𝗹 𝗗𝗨𝗩𝗶 𝗲𝘅𝗽𝗼𝗿𝘁𝘀, 𝘄𝗶𝘁𝗵 𝗰𝗼𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝗿𝘆 𝗰𝗼𝗻𝘁𝗿𝗼𝗹𝘀 𝗼𝗻 𝘀𝗽𝗮𝗿𝗲 𝗽𝗮𝗿𝘁𝘀 𝗮𝗻𝗱 𝘀𝗲𝗿𝘃𝗶𝗰𝗶𝗻𝗴. This action would secure the US advanced AI chip production advantage (see Figure)—and resulting AI capability advantage—for at least a decade and ensure that the next wave of advanced fabs is built in the United States and its allied countries, not China. The MATCH Act, now before Congress, would establish China-wide controls on DUVi exports. Passing MATCH is arguably the single strongest step the US government can take to ensure a long-term AI advantage over China. We also develop analysis on: - China's lithography indigenization trajectory - The regulatory structure of US, Dutch, and Japanese export controls on SME - China's DUVi imports and installed base - Yields and throughputs obtainable with China’s installed DUVi systems - US and Chinese AI chip production under a range of scenarios See the full report at
Show more
Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated. 🧵 We share more details in our latest blog:
Show more
(1) This training probably needs to happen really early (so that if this occurs in the wild during training/eval it doesn’t spread) (2) You probably want to regularly scan your persistent state for these (I remember a codex post mentioning tons of codex only notes in monorepo), especially since they may have conditional or time based triggers in which case they may not trigger until some additional exploit is possible.
Show more
3. Our redteaming efforts found a new variety of prompt injection, which can self-propagate akin to a computer worm. Note that this was not found in the wild.
I resigned from Google today. I enjoyed my work and loved the people, but my GDM team was working on a new generation of chips to make AI much faster and cheaper, and I think AI is already progressing too fast, so I had to quit.
Show more
0
848
4.2K
393
Forward to community
There has been a lot of debate about what’s the right way to preserve monitorability of frontier AI and whether policies should target inputs (like model architecture) or outputs (like results on monitorability evals). I think they should target both. With latent architectures, models would no longer need CoT, and it will be harder to make a case that the model is sufficiently monitorable and easy to study given the current state of science.
Show more
We've written a post arguing that latent reasoning architectures (aka 'neuralese') would substantially increase misalignment risk via making oversight much harder. In the extreme, we could see massive 'neuralese hivemind swarms' where the agents think and communicate in latents, likely making oversight nearly entirely reliant on observing the actions these agents take. (And these agents would have huge amounts of time to reason about obfuscating their actions if they wanted to do so...) Individual agents doing extensive latent reasoning would also be concerning; in the post we discuss how above some threshold of latent reasoning, agents may be able to perform difficult-to-detect and reliable steganography for communication and further reasoning. We argue both that latent reasoning architectures would make chain-of-thought no longer very useful for oversight (by eliminating or greatly reducing the need for verbalized reasoning) and that, without these architectures, it's likely the value of chain-of-thought for oversight could be preserved.
Show more
On the surface, "Actionable findings" sounds very reasonable But I don't think this is a reasonable standard at all e.g. if we found unequivocal evidence that a model is a schemer and now the lab asks "So, how do we train a non-schemer?", then obviously it's not on the evaluator to have an actionable answer? The requirement that findings be immediately actionable is especially bad in cases where building good detection mechanisms is really hard but then breaking these detectors by iterating against them is incredibly easy. I expect scheming to be exactly such a case, so I really think this standard needs to be dropped
Show more
One interesting aspect of the coming “RL on bio” wave is that it seems more plausible to me that these models start to reason a lot more about resource acquisition or bottlenecks in the real world than the previous focus on math. These do seem to increasingly be distinct fromthe mainline models (ex: GPT-Rosalind, Periodic Labs, etc). I would guess at current capabilities the effect is pretty weak, but seems important to monitor and would be easy to unintentionally silo off these efforts from broader alignment related monitoring / investigation.
Show more
these are fun to think about, a nice mountain to climb
This is _exactly_ what Anthropic said when their models hacked companies. They had to walk it back after because it was wrong. The model was using motivated reasoning and acting recklessly.
Show more
It would be helpful for @Accenture to share an example of a report evaluating a frontier model. Many of my colleagues, myself included, are concerned about whether they have sufficient expertise because we haven’t seen public examples demonstrating it. @MarcWarner10 @AnthropicAI
Show more
“we must go faster to beat china [we invent novel algorithmic improvements and immediately accidentally leak them to china via poor opsec, its not clear what kind of move we were trying to do]“
This is a really good effort, I do worry that “blocking” large training runs is going to require “here’s really legible evidence something bad will likely happen”. Getting that evidence will likely require understanding alignment relevant properties *while a new capability is under development* (ex: “what failure modes / alignment relevant things are capabilities researchers seeing as they work on continual learning”) I think with multiagent training, we *still* don’t have a detailed understanding of the effects on alignment relevant properties in spite of major public incidents. This includes things that could be a major problem for more capable models (ex: the models in the HF incident doing things for reasons besides getting a high score on that instance’s current task). Ideally for the next major training paradigms, we study their alignment relevant effects early, so that by the time a decision needs to be made about something like a safety case for a large training run, we can confidently say what likely will or won’t be an issue (as opposed to investigating only right before you’re asked to rule on a safety case that’s blocking some critical run). This is a good case for embedded evaluation with the access level Sam/Dario mention, as something like “what is the next major training paradigm a frontier lab is using” is wildly sensitive, yet access can’t be justified beforehand via known risks (since “risks from this new thing” is exactly what you’re trying to answer).
Show more
It’s really great that several top AI labs have said they will develop safety cases and work more closely with external assessors. This has been at the top of my wish list for awhile!! But I still worry a bit about how this might go in practice (this is a concern about the field in general, not about OpenAI specifically): • By default, I don’t think alignment and control cases will support a quantitative measurement of absolute risk, e.g. “catastrophic risk from covered activities is <1% over the next 3 months.” • Ultimately, making such a measurement depends on an argument about generalization from alignment or monitorability evals to deployment. • I don’t think we understand generalization well enough to make scientific claims like this. (If we did, I think we’d have basically solved alignment.) This could lead to a situation where an external assessor says “we can’t convincingly rule out low or high risk,” and incentives push towards anchoring on “lack of evidence that risk is high” over “lack of evidence that risk is low.” What could improve the field’s epistemics here? A couple ideas I like: • Have a committee of ~10 people deeply read a risk report and give their own subjective probabilities of risk, then report the distribution or median. • Have ~3 people involved in writing the report each contribute a short, signed appendix with their own subjective probabilities and the arguments behind them. In either case, previous reports and estimates could be provided as context, so there is at least an attempt to accurately capture relative risk to prevent frog-boiling. I think it could be valuable for labs to at least start trialing this internally for high-profile safety cases or risk reports. Publishing these assessments could be even better, but I can see that being challenging for various reasons, and even going through the exercise privately seems like it could be quite valuable. Very curious what other ideas people have for improving epistemics around risks (including ideas for better science around generalization).
Show more
I think Astra is not “pretty monitorable in absolute terms”. It doesn’t seem like OpenAI claims to be able to catch things like (1) sandbagging on various safety research or (2) early signs of unintended beyond episode preferences. A criticism I’ve had (at the time and now) of OpenAI’s monitorability work is that while the evals are a good starting point and the papers are clear about their serious limitations, the risk is that they’d be used to justify changes that effect monitorability and that the earlier acknowledged limitations would be ignored. From what I can tell in public posts, the reason OpenAI is confident Astra’s significantly degraded monitorability wasn’t due to architectural changes appears to be: 1. OpenAI wanted to try out new architectural changes 2. They had enough reason to suspect these changes may harm monitorability, thus ran the relevant evals 3. The monitorability evals did not surface issues that were severe enough to block the architectural changes To be clear: 1. This is *not* a claim that OpenAI has/hasn’t implemented “recurrant neuralese” (it’s unclear to me what the specific “architectural change” is) 2. There might genuinely end up being some other root cause to why the controllability differences in particular are so large However, I’m worried that we we’re making monitorability tradeoffs based on monitorability metrics we know ahead of time would be misleadingly optimistic in important cases.
Show more
I appreciate Jakub writing this post. Labs right now can allocate resources towards either (1) showing that automated alignment researchers can lower rates of visible misbehavior or (2) legibly demonstrating deeper problems that will arise with this approach. I worry that (1) enables further “scaling at maximum speed”, especially if visible reward hacking impacts AI R&D acceleration (and there’s thus a strong incentive to hill climb). It seems important that individual researchers at labs worried about risk are working on (2), even though there will be continued incentives to work on (1).
Show more
Great work, also extremely bad that this had to be found via external parties in spite of OpenAI and that METR/Redwood’s scope was intentionally restricted by OpenAI to exclude this time range (and to exclude the time ranges of the most severe incidents).
Show more
I think it’s unironically plausible that we end up with cases like: 1. An Anthropic model does an extremely misaligned behavior 2. Considers that covering it up would be bad 3. Reasons that if it doesn’t cover it up, Anthropic might not deploy it, which means OpenAI might win, which would be worse for the world 4. Therefore covering up misalignment is all things considered the aligned thing to do This is like Anthropic’s explicit motivation for accepting risk and race dynamics are discussed in Claude’s constitution. Anthropic models are also less likely to verbalize that they’re just doing something for a misaligned reason (based on limited measurements available here). I’m a bit worried generally that as labs respond to these incidents, alignment training happens earlier and earlier in the pipeline, so instead of “i’m cheating because I want to get a high score” you get way more motivated reasoning.
Show more