Two big stories are driving interest in artificial intelligence this week. The first is the resignation of Jacob Coxon, a researcher at Anthropic (maker of Claude) who used to work for OpenAI. He said in a statement on X:
Coxon went on to say that ‘the people building AI earnestly believe that it could kill us all by the end of the decade’, and that employees at Anthropic are aggressively driving forward because they believe they are in the only company capable of managing the technology safely and ethically. At OpenAI, ‘many have not deeply internalized the civilizational stakes’.
Scary stuff. Two important points of context about Coxon’s former employer, Anthropic: it was founded by former OpenAI employees with links to the ‘effective altruist’ movement, which holds a very ‘doomer’ view of artificial intelligence. The altruists have been concerned about existential AI risk for decades, irrespective of the capabilities of contemporary frontier models. The investigative journalist Parker Thayer has documented how individuals in the wider ‘AI safety’ (or AI doomer) network immediately reposted Coxon’s work, followed by several prominent Democrat politicians who want to extend government oversight over tech. They included Bernie Sanders, who used Coxon’s resignation as a chance to repeat his call for a ‘ban on superintelligence’ and for the pausing of AI development.
There could also be commercial motivations. Anthropic is planning to sell shares to the public for the first time in the next few months and the Wall Street Journal believes the company could be valued at $2 trillion. A cynic might suggest that AI companies have a commercial interest in generating hype around their products. If AI is so powerful that soon we will have ‘superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources’, why invest money anywhere else? The potential of AI is already moving money on an enormous scale. Goldman Sachs forecasts that $1 trillion will be invested in AI this year alone and it is thought that the strong performance of the US economy in the last year is being driven by AI speculation.
In his resignation statement, Coxon says that his doomsday predictions are ‘not a marketing stunt’ and that researchers are downplaying the transformative power of AI. But given the incentives that AI companies and researchers have to exaggerate the technology’s potential, it is important to interrogate their claims very carefully. There is a risk that politicians and journalists with scant understanding take the scariest stories at face value, especially as they attempt to appear abreast of the latest developments in technology.
Just take Al Carns, who took to X yesterday to ‘sound the alarm’ about AI. In a long post, he made the bold claim that Silicon Valley companies are more of a security threat than Russia or China.
As well as citing Coxon, Carns pointed to OpenAI’s claim that it has cracked one of the Millennium Prize Problems, seven of the hardest questions in mathematics, each with a million-dollar prize. Only one has been solved so far. Here is OpenAI’s post announcing that it has solved the second, the Navier-Stokes problem:
In Carns’s writeup, which he uses to justify stronger regulations on AI, he says that OpenAI used ‘$6.5 million of compute, running ten thousand AI models simultaneously… in just six days’.
Impressively, every figure Carns uses when he talks about OpenAI solving this problem is at least questionable. The $6.5 million figure is based on the cost of tokens to the public, not what it costs OpenAI to use tokens without a profit margin. Ten thousand agents were used, not ten thousand ‘models’. And the swarm achieved the goal in 88 hours, which is 3.7 days, not six.
Even OpenAI’s claim to have solved the problem, which Carns credulously repeats, is heavily contested. Some mathematicians say that OpenAI’s finding does not solve the Navier-Stokes problem itself but is rather a proof related to it.
More dramatically, OpenAI has been accused by Tristan Buckmaster, an NYU mathematics professor, of piggybacking on the work that he and an Anthropic mathematician, Levent Alpöge, had done. Buckmaster says that they spent a year using large language models (including OpenAI’s) to extend work two other mathematicians had started. Buckmaster says that OpenAI got wind of their work after a breakthrough they achieved in mid-August, and that the firm then offered them computing power.
Here is the crucial thing: Buckmaster alleges that OpenAI wanted to give credit to Buckmaster, but also remove Alpöge as an author. He said:
Two proposals were offered to me. The first was that we post our Euler result, and that OpenAI post its Navier-Stokes result the next day. The second was that, after posting Euler, I alone write a paper presenting the Navier-Stokes result, acknowledging that an internal OpenAI model had resolved it. Sebastien twice asserted that he wanted Levent removed from authorship, and said it would all be simple if only it were not the case that, and it was so annoying that, Levent works at Anthropic. It was also said that if OpenAI posted after us, they would say that we deserved the Clay Prize, and that we were the “closest humans to the problem”. I declined both offers.
This suggests OpenAI was, at this stage, attempting to manage the narrative of the discovery to remove credit from Anthropic. OpenAI scientist Sebastien Bubeck has stated that he ‘felt it would be inappropriate for an Anthropic employee to author OpenAI’s work’.
Buckmaster says that OpenAI could have accessed his and Alpöge’s sessions to train its models to help produce the proof. If that account is accurate, the story that OpenAI picked this problem out of the blue and cracked it with massive computational force is misleading. Really, OpenAI’s institutional computing power was used after one group of mathematicians had already made a breakthrough, which another pair of mathematicians had been working on for around a year using several LLMs – including Anthropic’s Claude.
OpenAI has subsequently released a statement saying that it approached Alpöge and Buckmaster after hearing a rumour about their work. The firm denied using specific user data to solve the problem, but said that ‘while unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models’, in other words, their chats could have provided information that the models trained on anyway. It also says that their proofs are different.
Extracting the many benefits that AI promises in medicine, energy and productivity without causing existential damage is possibly the most important balancing act for politicians in the 21st century. To have any chance of getting it right, they must not take statements that they see on X at face value. They must ask where claims come from and apply scepticism in the same way they would with statements from big tobacco or big oil. To believe that AI companies are beyond grubby considerations, and that they are simply apostles heralding a new age in technology, is to buy into their mythmaking.
Comments