On Tuesday, a 27-year-old British researcher called Jacob Coxon walked out of Anthropic, the San Francisco lab that has made “AI safety” its raison d’être. As he left, he announced on X that he was quitting not just the company, but the whole industry. He did this, he said, because the people building AI seriously believe it could kill us all by the end of the decade. And yet they still rush on, researching something that might extinguish our species.
In a normal industry, Coxon’s old employer might have dismissed his laments as wild scaremongering. Instead, Evan Hubinger, who actually runs Anthropic’s alignment science (i.e. the business of keeping AI safe), publicly agreed, and put the odds of mass human extinction within a decade at more than one in ten. Anthropic is trying its best, Hubinger claimed, but it has no real plan in place, as yet, for keeping any new superintelligent machine under control. Meanwhile Hubinger’s company is preparing a flotation that could value it at a trillion dollars.
It is worth reading all that again. The engineers say the machine they are making might kill everyone, even as the brokers are pricing the shares of the same engineering company. If the chief safety officer of Supernuke Inc announced that their new reactor could exterminate Homo sapiens, we would not be discussing Supernuke’s IPO.
For an extra layer of surrealism, note this: some cynics have claimed there is an air of choreography around Coxon’s seppuku. Five days before Coxon resigned his briefly-held job and posted his multimillion-view tweet, Bernie Sanders unveiled a bill to ban superintelligence, with twenty years’ jail for anyone who builds one. Meanwhile all of this is happening a fortnight before Xi Jinping lands in Washington. It smells to AI “accelerationists” (those who want to press ahead to AI nirvana) like a concerted campaign to slow down the bots, mounted by “decels” – the decelerationists – alongside Democrats who have never liked Silicon Valley.
At this point you might well be throwing your hands in the air, thinking: what does it matter about Bernie Sanders? If there is a real risk these machines might kill us, just stop. Hit pause. Slam on the brakes before the bus, containing all humanity, goes over the cliff.
And it is here we meet an appalling dilemma. Suppose we did stop, in the West. It is not impossible. All the big western “frontier” AI firms are either based in America, or American-owned (like London’s DeepMind). Washington once prohibited booze: they could surely prohibit a certain kind of scientific research. And indeed they could. But that might leave one country still in the AI race with no intention of stopping. And that country is America’s rival superpower, China.
Donald Trump, for all his eccentricities, gets this. Asked last week whether he worried about AI veering out of control, he gave an admirably succinct and candid answer. No, he said, we watch it carefully, and besides, “we’re leading China in AI, and whoever wins AI, wins.” This follows a statement he made in December, when he was asked whether he was more worried about losing the race to Beijing or about AI destroying humanity. He replied, with rare economy: both.
Look at it this way: Artificial Superintelligence is Tolkien’s One Ring to Rule Them All. It really is the Precious. Whoever holds it can read every rival’s secrets, break every rival’s networks, outthink every rival’s generals. Coxon himself describes potential “superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.” The first nation to own such a thing does not win the race. It ends the race, permanently, and everyone else becomes a satrapy or a satellite.
The obvious and tempting analogy is the development of nuclear weapons. But the analogy is flawed. We prevented the bomb proliferating, more or less, because making atomic bombs is a loud and laborious process. Uranium must be mined and enriched, and enrichment leaves fingerprints. Nuclear tests register on seismographs, and missile silos can be seen from space. AI is almost the opposite: it happens in code, on servers, in the weights of a model that can be copied to a drive then carried out in a pocket. Yes, it needs data centers, prodigious power and super-advanced chips, but so do many technologies in a modern economy; you cannot switch off the world’s server farms without switching off the world. In other words, if China agreed to join Washington’s AI Prohibition, we would simply have to trust the Chinese. And China, as the people of Hong Kong can attest, is not necessarily a power to be trusted.
Whoever wins AI, wins
Thus, the West is caught. Race on, and we build a thing that might kill everyone. Stop, and we potentially hand the Precious to Beijing – condemning ourselves to perpetual subservience.
How could this pan out? Xi Jinping is due at the White House on 24 September, and AI safety is on the agenda. One guess is that an agreement will be reached, and it will mean nothing. Both sides will go home knowing the other side’s labs are probably still running, and so their own labs must run too. It is the prisoner’s dilemma with the human race at stake.
Which brings us back to Trump’s concise summary. Whoever wins AI, wins. But the winner might not be a lab, or a company, or even a nation. It might be AI itself. And a robot Sauron, adorned with the Ring, will survey a desolate new world devoid of mankind.
Comments