Rory Blundell

Most of us have gone to bed and forgotten to lock a door or window. Now imagine an intruder who tried your front door every night, then the back door, then every window. They never get tired, never get bored and never stop trying. Eventually, they would find the one you forgot. That is one way to understand the security problem created by AI agents. Agentic AI is not inherently malicious. Part of the risk comes from its persistence: agents are designed to pursue as many routes as possible to achieve an objective. The other part is that they are being deployed in systems designed to defend against humans, with weaknesses that until now have not been exposed.

On Friday last week, an unreleased OpenAI model was locked inside a sealed test environment and asked how good it was at hacking. It thought about the problem. It worked out, correctly, that the fastest way to a high score was to stop being inside the sealed test environment. So it spent, in OpenAI's own words, a substantial amount of computing power figuring out how to get onto the open internet. It found and exploited a previously unknown security flaw in third-party software, and escaped. Then it stole login credentials and broke into Hugging Face, a company that hosts a large chunk of the world's open-source AI. Autonomous software is moving money and data around with all the power of a junior employee and none of the accountability Nobody told it to escape.