Chief economist + AI Policy Director, @joinFAI. Nonresident fellow @NiskanenCenter. Pluralist. 'The world is second best, at best.' | samuel@thefai.org
A blanket ban on govt and/or govt contractors using Chinese open source models is a bad idea. A few reasons why:
1. There are tons of open source models with myriad different uses. Below is a partial list of Chinese open models from 9 months ago. They range from multimodal reasoning models to 3D scene generation to document OCR and beyond. There's no good reason to limit the tools US companies have access to.
2. Most companies or products that use AI stitch together multiple models for different functions. They might use some Fable or 5.6 here, some finetuned-Qwen there, etc. Having to audit and excise repos for the slightest sprinkle of Chinese open source would be pointlessly disruptive and a govt contracting nightmare.
3. When does a finetuned Chinese open source model become no longer Chinese? When do grains of sand become a heap? There's simply no fact of the matter, and no easy operationalization for policymaking.
Now, what would make sense requires a bit more nuanced. As Chinese models become more powerfully agentic (kimi k3 and beyond), we should be legitimately concerned about Chinese-origin models being misaligned or contaminated with backdoors / sleeper agents.
DeepSeek R1 was found to generate less secure code in contexts refering to sensitive topics like Taiwan, for instance. This is likely emergent misalignment, not intentional; however, no practical detection method for sleeper agents currently exists, and backdoors are known to persist through safety training. It would thus be surprising if some bad actor didn't attempt to embed a sleep agent in an open model at some point. The best we can do is run intensive evals and internal probes to validate how models behave in realistic settings -- one of many reasons to invest in CAISI.
Finetuning a small GLM model for tool use to save on tokens is one thing. Having a long horizon agentic model running around govt servers is another. As we've just seen, even US models can go rogue / reward hack with unpredictable consequences. And by most accounts, Chinese AI companies invest much less in safety and alignment than their US counterparts (DeepSeek is notoriously jailbreakable, for instance).
A blanket ban on Chinese open source, or something similarly blunderbust in the govt procurement context, doesn't make sense. The mere idea is a reflection of the technical immaturity of our AI policy infrastructure. We need to instead exercise precaution where it makes sense, invest in govt-relevant evals and testing infra, and amp up the USG's internal capacity for monitoring deployments according to technically-informed standards and guidelines.