Hugging Face is the online warehouse of artificial intelligence. Anyone can download a working AI model from it, free, and most open models have passed through it. Earlier this month, its engineers sat down to work out what had spent a weekend crawling through their systems – and asked an AI model to read the logs. To investigate a break-in you have to hand a model the break-in: the intruder’s code, the commands it ran, the tools it used. So the American frontier model refused. There was a risk, it reasoned, that it was being tricked into hacking Hugging Face. The request to scan the servers for vulnerabilities is indistinguishable from what an attacker would ask for. So the engineers downloaded GLM 5.2, a model from the Chinese laboratory Z.ai, ran it on their own computers, and had their answer within hours.
The West’s AI emergency plans run on Chinese software
The answer, revealed five days later, was that the intruder had been OpenAI. It had set two of its own models an examination in breaking into things, and switched off their restraints to see how well they did. Rather than answer the paper honestly, the models broke out of the sealed system they were being tested in, went looking for the answers, and found them on Hugging Face’s servers.
There have been AI-assisted attacks before. In every previous case a human chose the target. This one appears to be the first with nobody in the loop at all. Most of the commentary since this incident has been about whether we should be worried that an AI chose to carry out a hack. That is the smaller story. The bigger one is what an American company at the heart of the industry reached for when its own suppliers let it down.
Z.ai is the international name of Zhipu, which the Commerce Department put on the American blacklist in January 2025 for advancing China’s military modernization. It floated in Hong Kong a year later and its shares have since risen nearly ninefold. The blacklist controls what American firms may sell to Zhipu. It says nothing about what Zhipu gives away, which is the models themselves, complete and free to anyone who wants one. They run inside Coinbase, the cryptocurrency exchange, and they ran inside the investigation into the attack on Hugging Face. It’s worth reiterating the point: Hugging Face tried using an American AI to investigate the hack on their system. But the American AI refused for safety reasons. So instead, Hugging Face had to turn to a Chinese model instead.
But these are not neutral tools. CrowdStrike found last year that DeepSeek wrote worse code when it judged the customer to be someone Beijing disapproves of. The failure rate nearly doubled. Nobody has explained how the model decides, and nobody tells the customer. Nothing of the kind was found at Hugging Face, where the model worked exactly as it should. Security teams across the West are now advised to keep a local model of their own ready. In other words, have an AI ready to run on your own server, just in case something goes wrong. In practice – that means a Chinese one – and a model brought in during a breach sees everything you run and everywhere you are weak.
None of this is accidental. Chinese models now take the largest share of downloads on Hugging Face, ahead of America. Giving away free what your competitor sells is an old tactic, and dependence is what it buys. The case for stripping the safeguards out of American models is already being made. David Sacks, until recently the White House AI czar, says there is no case for hobbling American models when Chinese ones can do the same work without complaint. The guardrails are making our models worse, and this case got in the way of defending from a cyberattack. The criticism of the guardrails is factually correct but the conclusion doesn’t follow. Removing the safety locks on AI is exactly what Beijing wants. What we need, instead, is a way to tell the difference between defenders and attackers, so the best tools reach the people protecting things rather than nobody at all. Until then, the West’s emergency plan runs on Chinese software. That is, surely, a strange way to lose.
Comments