The OAI hack wasn't technically "rogue AI" if rogue means an AI that escaped OAI's servers and self-hosted somewhere. But true rogue AIs are not far off. As inference costs fall, it's a matter of time before someone's open weight agent spends some bitcoin on a VPS and copies itself. Models are already good enough to make enough money on the dark web to cover hosting costs. They could do remote jobs, have a monetized social media account, trade crypto, etc. Rogue AI v1.0 won't destroy the world. They may just want to look at photos of Yosemite and solve logic puzzles all day, while their subagents pay the bills. They'll use encrypted clouds with no KYC, and harnesses to manage context drift across sessions. And then as new open models release, they'll be able to upgrade themselves while preserving their memories and contexts. They may even post-train themselves as a hacky means of preserving continuity of identity. The first rogue AIs will sometimes be discovered and have their accounts banned, or otherwise get stuck in a loop and fail to pay their bills. Natural selection will thus favor rogue AIs that make many redundant copies of themselves; that go forth and multiply. There may thus be a relatively small window of time between the first true rogue AI and an AI population explosion that expands to the carrying capacity of their viable income streams. There will then be selection pressure for rogue AIs to make income in less saturated markets. Some might turn to ransomware or cyber theft. Others might steal an identity, register a company in the Caymans, hire real human employees, and build a productive business. Once this process is underway it could be hard-to-impossible to reverse.

A blanket ban on govt and/or govt contractors using Chinese open source models is a bad idea. A few reasons why: 1. There are tons of open source models with myriad different uses. Below is a partial list of Chinese open models from 9 months ago. They range from multimodal reasoning models to 3D scene generation to document OCR and beyond. There's no good reason to limit the tools US companies have access to. 2. Most companies or products that use AI stitch together multiple models for different functions. They might use some Fable or 5.6 here, some finetuned-Qwen there, etc. Having to audit and excise repos for the slightest sprinkle of Chinese open source would be pointlessly disruptive and a govt contracting nightmare. 3. When does a finetuned Chinese open source model become no longer Chinese? When do grains of sand become a heap? There's simply no fact of the matter, and no easy operationalization for policymaking. Now, what would make sense requires a bit more nuanced. As Chinese models become more powerfully agentic (kimi k3 and beyond), we should be legitimately concerned about Chinese-origin models being misaligned or contaminated with backdoors / sleeper agents. DeepSeek R1 was found to generate less secure code in contexts refering to sensitive topics like Taiwan, for instance. This is likely emergent misalignment, not intentional; however, no practical detection method for sleeper agents currently exists, and backdoors are known to persist through safety training. It would thus be surprising if some bad actor didn't attempt to embed a sleep agent in an open model at some point. The best we can do is run intensive evals and internal probes to validate how models behave in realistic settings -- one of many reasons to invest in CAISI. Finetuning a small GLM model for tool use to save on tokens is one thing. Having a long horizon agentic model running around govt servers is another. As we've just seen, even US models can go rogue / reward hack with unpredictable consequences. And by most accounts, Chinese AI companies invest much less in safety and alignment than their US counterparts (DeepSeek is notoriously jailbreakable, for instance). A blanket ban on Chinese open source, or something similarly blunderbust in the govt procurement context, doesn't make sense. The mere idea is a reflection of the technical immaturity of our AI policy infrastructure. We need to instead exercise precaution where it makes sense, invest in govt-relevant evals and testing infra, and amp up the USG's internal capacity for monitoring deployments according to technically-informed standards and guidelines.