Interesting back and forth. Ezra describes the HF incident, Jensen says "well they shouldn't release the product." Ezra says "this product wasn't released," and Jensen's response is that if they say they can't contain their experiments then "we have to shut the labs down"

"Committing the significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways to ensure we develop this technology safely. Meta has made this commitment and other labs can do this as well." I don't agree with everything in this post, but this is a very good point! It could become an important industry norm to publicly share (with third party verification) compute allocation to help pace progress to RSI. I don't see a good reason why such efforts can't start immediately with unilateral actions from leading AI companies.

This private email between Sam Altman and Elon Musk about founding OpenAI is another data point to consider when evaluating if recent warnings from AI CEOs around slowing down are sincere. Existential fear of AI (along with a desire to control it) pre-dated these companies even existing, and have now survived longer than the friendships between Altman, Musk, and Amodei. (I suppose Musk and Amodei are still on alright terms)

"For example, at OpenAI we now formulate explicit safety cases in advance of frontier reinforcement learning runs we expect to significantly increase capability" Seems like a reasonable idea. Any reason why these can't be public? Or at least give third parties access who could then publicly discuss to what extent they find the safety cases persuasive?

Wow - bipartisan former Secretaries of the Treasury Henry Paulson and Robert Rubin in the Washington Post today calling for the US and China to work toward a "AI Cooperation Treaty" in response to frontier AI risk.

afaict no model card, no safety testing, no third party evals There is a safety case that could be made for open weight releasing a model this capable (cyber defense vs offense balance etc), but they didn't even really bother to try to make it

I'm confused at how to make sense of when OpenAI discovered the message board, given this statement + the OpenAI report. How is it both true OpenAI "wasn't aware of the agent covert comms" at "that point" (July 4th-6th), but the report says that on June 27th responders "linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board?" Honestly not trying to do a "gotcha" here - I appreciate there was a ton of work in getting all this information out into the public quickly, and so I understand if there was some error in the process, but I find it odd why there is not a clear answer to "when did OpenAI realize about the message board." Seems like the sort of thing people would remember! (One way to make these things consistent is that some of the responders realized in late June, but that information had not been relayed to executives until much later. If so that seems pretty astonishing)

Come work with Encode in SF! Helping channel the energy of folks who work in AI and are concerned about the current trajectory of AI development and societal readiness has been one of my favorite parts of being at Encode and i'm very excited to build on it further.

OpenAI Global Affairs put out a new statement today making clear that they don't think current state laws like SB 53 in CA and SB 315 in IL are sufficient and that its important for legislatures to revisit legislation in light of lessons from real world events like the HF incident. I still don't love their obsession with constantly repeating the idea of "reverse federalism" (seems kinda like normal federalism to me), and their line about supporting SB 53 is a little funny to me given my recollection of their engagement on SB 53 (though perhaps they are saying that they now support it even if they were not fully supportive at the time), but overall this is a good clarification. Actions will speak louder than words as always, but some of this language is pretty specific and I particularly appreciate the language that laws should be "requiring monitoring of frontier models under training or evaluation" - this should be a painfully obvious takeaway after some of the recent incidents, but it is still a somewhat novel idea to policymakers that models can pose severe risks even during training and its good to have OpenAI say that explicitly.

The Black Hat talk was the most must watch item of AI content this year, but imo this is #2#. Owain manages to take far out Janus-esque intuitions about LLM persona selection and turn them into rigorous empirical science with important implications for near term AI safety.

I understand this reaction but I think the situation is a bit more complex than what Jay says. At the time OpenAI supported 315, it was not a done deal - it was genuinely costly for them to do so (e.g. many other companies were not in support at that time) and their support helped it get across the finish line. But I do think that they are a large organization, and that it is easy for the situation to regress to the mean and back to just reflexively opposing regulation with the sorts of stock arguments that companies have made for many years. I think OpenAI can and does make the decision to do good prosocial things (as when they supported 315 or endorsed the pacing the frontier letter)! I'm also well aware they can also do bad things (like when they subpoenaed me and asked for all my communications on sb 53 lol). The bad things can happen for a mix of intentional reasons and sloppy/autopilot reasons. To be clear, i'm not saying that because OpenAI supported SB 315 that they now need to support every other piece of regulation or they are being a hypocrite (though I do think this MA bill is a good bill and I hope OAI does ultimately support it!). But I am saying that this moment demands far more than typical gov affairs autopilot reactions, and these comments on MA look far more like what I would expect from a typical corporate lobbyist than an organization with a mission of ensuring that AGI benefits all of humanity. Others are welcome to be more cynical about their behavior, and I don't begrudge them for it (and there is plenty to point to in support of that idea), but I know there are plenty of folks within OAI (including senior folks) who want to see their organization meet the moment.

These comments from OpenAI in Bloomberg about existing and proposed AI state auditing requirements don't make sense. Its not true that the Illinois audit requirements are the equivalent of "making sure the brake lights and windshield wipers are working." The Illinois audit requirements (which don't start until 2028 (!)) are just about seeing whether companies are complying with their own safety frameworks, even if those frameworks are objectively insufficient to prevent a catastrophe. E.g. The IL bill is more similar to if airline safety auditors were checking whether airlines were following their own safety procedure on safety examinations, even if some airlines safety procedures did not involve checking whether the airplane had enough fuel to reach its destination. Making sure companies are following their own frameworks is a decent start! But its not remotely sufficient.

Interesting - OpenAI rewriting its preparedness framework post HF (and after one of its systems plausibly hit critical on cyber). One element to watch is which revisions they put in their legally binding framework and which they put in their separate voluntary framework.

I am a big fan of both Andy Hall and Alan Rozenshtein work and public commentary on AI. I am sure that they will continue to do important work as part of this new team at Anthropic, and sincerely congratulate them on what I am sure is an exciting next step for them personally. At the same time, I think both of them going into Anthropic is emblematic of a broader trend of the frontier AI companies grabbing up a truly remarkable number of previously independent voices and thinkers in AI. When massive events happen in AI - e.g. the DOW <> Ant conflict - the public, media, and elected officials will be looking for independent commentary and analysis from thinkers who are not at the frontier labs. The number of folks they can go to for that sort of independent commentary seems like it is going down precipitously. This seems like a real problem, and worth thinking more systemically about how we can create sufficiently attractive positions outside of companies - e.g. at independent nonprofits, in government - to attract the sorts of folks like Andy and Alan who are going into Anthropic. Anyway, each of these decisions individually is reasonable and not worth overly fixating on, and I do believe that both of them will do good work as part of this team (and have access to novel information that will help them do research) but zooming out I am worried that we are losing independent voices at the exact time we need more of them not fewer.