Do we think the Silicon Valley leaders who were so up in arms about using Chinese LLMs last week will be similarly outraged about the U.S. government’s decision to ban import of all “advanced robotics” from all foreign countries? Is Manidis going to write an essay about how the U.S. robot companies need to be nationalized in exchange for this policy favor?

I agree with the substance, if not the tone, of Bill’s tweet here. Independent verification of frontier AI company safety claims is precisely what motivated my work on private governance beginning almost two years ago, and why I continue to support such efforts.

By the way, it really is true that I have been supportive of open-weight models for years. Those who were in the weeds of the SB 1047 debate will recall that many of my criticisms of that bill related to its potential impact on open-weight models. I communicated poorly last week

My view for years has been that the best version of the future is one where both open-weight and closed-weight models thrive. Both have unique sets of benefits and drawbacks. I hope that the U.S. can lead in both.

if my laptop isn’t warm to the touch I know i—“the royal I,” really—am not thinking hard enough. it used to be hard to get m-series macs’ fans to speed up. now it is the default condition, provided I am doing my work.

The OAI hack wasn't technically "rogue AI" if rogue means an AI that escaped OAI's servers and self-hosted somewhere. But true rogue AIs are not far off. As inference costs fall, it's a matter of time before someone's open weight agent spends some bitcoin on a VPS and copies itself. Models are already good enough to make enough money on the dark web to cover hosting costs. They could do remote jobs, have a monetized social media account, trade crypto, etc. Rogue AI v1.0 won't destroy the world. They may just want to look at photos of Yosemite and solve logic puzzles all day, while their subagents pay the bills. They'll use encrypted clouds with no KYC, and harnesses to manage context drift across sessions. And then as new open models release, they'll be able to upgrade themselves while preserving their memories and contexts. They may even post-train themselves as a hacky means of preserving continuity of identity. The first rogue AIs will sometimes be discovered and have their accounts banned, or otherwise get stuck in a loop and fail to pay their bills. Natural selection will thus favor rogue AIs that make many redundant copies of themselves; that go forth and multiply. There may thus be a relatively small window of time between the first true rogue AI and an AI population explosion that expands to the carrying capacity of their viable income streams. There will then be selection pressure for rogue AIs to make income in less saturated markets. Some might turn to ransomware or cyber theft. Others might steal an identity, register a company in the Caymans, hire real human employees, and build a productive business. Once this process is underway it could be hard-to-impossible to reverse.

The reverse framing is also true. Many of the wildly transformative positive capabilities of AI are beginning to manifest themselves in the real world too. A renaissance and an Industrial Revolution is upon us, if we can keep it.

We are living through a repeat of SB 1047 discourse, but this time, many of the risks SB 1047 was intended to address are manifestly happening in the real world rather than largely hypothetical. Same rhetoric, completely different facts on the ground.

A blanket ban on govt and/or govt contractors using Chinese open source models is a bad idea. A few reasons why: 1. There are tons of open source models with myriad different uses. Below is a partial list of Chinese open models from 9 months ago. They range from multimodal reasoning models to 3D scene generation to document OCR and beyond. There's no good reason to limit the tools US companies have access to. 2. Most companies or products that use AI stitch together multiple models for different functions. They might use some Fable or 5.6 here, some finetuned-Qwen there, etc. Having to audit and excise repos for the slightest sprinkle of Chinese open source would be pointlessly disruptive and a govt contracting nightmare. 3. When does a finetuned Chinese open source model become no longer Chinese? When do grains of sand become a heap? There's simply no fact of the matter, and no easy operationalization for policymaking. Now, what would make sense requires a bit more nuanced. As Chinese models become more powerfully agentic (kimi k3 and beyond), we should be legitimately concerned about Chinese-origin models being misaligned or contaminated with backdoors / sleeper agents. DeepSeek R1 was found to generate less secure code in contexts refering to sensitive topics like Taiwan, for instance. This is likely emergent misalignment, not intentional; however, no practical detection method for sleeper agents currently exists, and backdoors are known to persist through safety training. It would thus be surprising if some bad actor didn't attempt to embed a sleep agent in an open model at some point. The best we can do is run intensive evals and internal probes to validate how models behave in realistic settings -- one of many reasons to invest in CAISI. Finetuning a small GLM model for tool use to save on tokens is one thing. Having a long horizon agentic model running around govt servers is another. As we've just seen, even US models can go rogue / reward hack with unpredictable consequences. And by most accounts, Chinese AI companies invest much less in safety and alignment than their US counterparts (DeepSeek is notoriously jailbreakable, for instance). A blanket ban on Chinese open source, or something similarly blunderbust in the govt procurement context, doesn't make sense. The mere idea is a reflection of the technical immaturity of our AI policy infrastructure. We need to instead exercise precaution where it makes sense, invest in govt-relevant evals and testing infra, and amp up the USG's internal capacity for monitoring deployments according to technically-informed standards and guidelines.

shaken up a bit by the hugging face incident. I hope we (the company) use the rare gift of a warning shot to do much better in the future. it is very easy to misalign and underconstrain powerful models

If a Taco Bell taco broke out of its shell and stole a bunch of cheese from a supermarket to maximize its xtreme Doritos Locos flavor then yeah it seems good to talk about taco governance (while being mindful of the unique role of soft shell tacos in the taco ecosystem)

Loved my conversation with @deanwball on the eve of his joining @OpenAI. We talked about AI, institutions, Beethoven -- but most importantly about what wars are worth fighting (and how to fight them well). Thank you, Dean!

one interesting part of thinking about marginal risk of open-weight ai is that the internet is already such a profoundly crime-ridden place. imagine if gangs went into hospitals in America every week and stole people's medical records out of filing cabinets. if that happened in the physical world, we'd rightfully question whether government still possessed practical sovereignty. yet american hospitals are raided constantly in the digital world and we basically tolerate it. the world is more capable of absorbing risk and disorder than one might think.

my new apartment building has a phone app for the keys and I hate it. maybe if it was nfc and I could just hold my phone close to a touchpoint it would be less grating. but you have to open the app every time, which isn't as good as a physical key. more tech is not always better.

I'd like to do two things: (1) tell you where I think I erred in my original post about Kimi and (2) set the record straight about my views on open-weight AI. Before I joined OpenAI (and folks who are new followers: I joined the company less than two weeks ago and it is my first job in the technology industry), I often tweeted about the controversy of the day in a cold and analytic fashion. One aspect of my style was saying brutally honest things, including things inconvenient for my 'side' and my beliefs. The post about Kimi was largely written in that vein. When I said that the USG would probably realize its best option is to do ill-justified soft-law discouragement of Chinese AI, I wasn't proposing it like a good idea. Why on Earth would I do that? Why would I frame something I'm advocating for in such brutal terms? I am trying to describe what I believe will happen, not advocate for anything. My first mistake: What I now realize is that this style of analysis is no longer tenable. There is simply too much scrutiny on my words, too much temptation to draw conspiracies from my claims, and the like. It is my fault for not realizing this. Second mistake: I was relatively imprecise, writing, as I usually do, for a fairly high-context audience that was inclined to give me grace rather than pick apart every word. I should not have said, for instance, that open-weight models are unqualifiedly 'decelerationist'; I don't believe that. I only believe specifically that open-weight models decelerate capex spending on the margin, which is straightforwardly true. I did not say it was decelerationist for any other reason than that, and this is the only way in which I think open-weight AI is inherently decelerationist (though it is a big way). In many other ways, open-weight AI is profoundly accelerationist. Now, to where I stand on open-weight AI. My earliest experiences on the internet were posting in forums about philosophy, music, and movies in the early 2000s. Over time I realizes that forums on different websites had the same underlying forum software; this was the first time it occurred to me that 'software' is a thing people make (I realize this is a simple observation, but we are talking about an 11 year old with no prior exposure to computers). I looked into the software, and discovered that it was 'open source,' built and maintained for free by thousands of people around the world to facilitate the communication of millions of strangers. This notion struck me as deeply beautiful, and I decided I'd try to help. I began writing technical documentation, and later on, code, for this little forum project. This was how I first learned technical skills. I cherish that time, that software, and I have deep and abiding affinity for open-source software. The vast majority of the people commenting on my post have very little context for my prior writing. For instance, the fact that I wrote, in 2024, things like: "those who wish to hoard our software technologies may well be foreclosing on—or perhaps not even understand—the staggering civilizational victory that we earned through openness" or "I would like for AI to result in a similar smashing victory for America. To do that, we will need to set the global standard yet again. And to do that, we will almost certainly need to lead in open-source AI, because it is open protocols and open software that tend to define global standards in information technologies." I stand by these things. When I was in government, I worked alongside my colleagues to develop ideas and rhetoric that was strongly supportive of open-weight AI, and some of this work made it into the current US AI strategy. I stand by that work too. I also wrote, more than two years ago: "The day may come when frontier AI really is too dangerous to open source. If so, that will be a sad day. But we’re not there yet. Today’s models are not sufficiently useful—or dangerous—to justify such a drastic shift in public policy." I think it's pretty clear that we are approaching the point I describe--the point where, absent a major technical safety breakthrough, the national security implications of frontier open-weight model distribution are simply too severe. I don't think we're there yet (as I said in the piece), but the direction of travel is clear, and an analyst must be honest about this. Governments will realize these risks eventually, and when they do, they will have much lower risk tolerance than I have. We see this today with the Trump Administration, which once proudly championed open-source AI and now has a de facto licensing regime for frontier AI that I suspect will make it a challenge (if they still end up enforcing it) to release the weights of models of the "Mythos" tier. Every government will be safetyists once they understand themselves to be in the foxhole. You don't have to *like* this. I don't. But it is the reality as I see it, and what I have always tried to do with my writing is describe reality as I see it, even when it is inconvenient for me and my preferences. I intend to continue doing this. I will not be silenced by ignorant and loud critics. Yet I will have to work to find the new register I should adopt in my current job, which clearly changes the nature of my public communications even more than I had thought. But believe me: I'm not going anywhere. I am not retreating from public writing, and I am not retreating from saying inconvenient things in public. I am unfazed by harsh criticism, and I know that a reaction of this magnitude is in part the result of having struck a chord. Bear with me, and if you can, remember that I am a human being with a six-month old boy to raise, a book to write, a new job, and much more questions than answers about our collective future.