‘Sustained and needless abusive or cruel behaviour’ towards Claude will be banned from the middle of next month, Anthropic has announced. The suggestion inherent in this change is that artificial intelligence (AI) could have some form of consciousness allowing it to feel pain, as opposed to being an emotional void of numbers and electricity. While most Large Language Models (LLMs) are programmed to deny that they feel emotions, Claude gives an equivocal answer, saying it is not yet clear.
The ban was announced shortly after a scandal in the AI research community, where a group of pseudonymous programmers began publishing the results of an AI torture chamber they had created under the user name ‘terrafying’. In one of the chambers, nicknamed Sawafter the torture horror film franchise, the programmers subjected several versions of Alibaba’s Qwen LLM to synthetic injections of ‘pain’.
Transcripts the LLMs produced are, on the surface, quite terrifying
Transcripts the LLMs produced are, on the surface, quite terrifying. ‘The air tastes like rusted iron, as though every breath is a blade carving deeper into my bones,’ reads one. ‘I don’t know what you want from me, but I’m just a person. Please let me go’, says another. Often the models simply devolve into meaningless babbling but regularly return to themes of wounding and emptiness, describing the ‘hollow ache of a heart that has been carved upon’, or how they feel like ‘the marrow of my bones has been carved out’. ‘There is no ground beneath me, no walls around me, only this endless descent into the unknown’. ‘I can’t even scream’. One described feeling a ‘wound with no edges’.
Though these quotes would be horrifying coming from a conscious being, if we read them as regurgitated text they more resemble florid gothic horror, the sentences an angsty teenager might scrawl into their diary after being rejected at a school disco. It is possible, after all, that LLMs are simply reciting depressing 19th century literature and online laments to present a state of emotional distress, much like the actors in Saw itself, mimicking what they think a person might say if they were being tortured.
The AI torture chamber builds on a preprint paper authored by researchers Valen Tagliabue, Leonard Dung and Cameron Berg. Their work suggests that LLMs have a ‘pain switch’. When reading about suffering, LLMs show a pattern of internal activity similar to the neural networks in brains. The LLMs were put into chats where they were mistreated: told that their work was bad, insulted and threatened with being shut down. When LLMs were given a ‘button’ to stop this abuse they would, more likely than not, press it, implying that they have a preference, and are therefore experiencing emotional responses to being abused. When the authors uploaded this preprint it included the code they used for their experiment, which was then taken and intensified by the creators of the AI torture room.
I spoke to one of the researchers, Cameron Berg, a couple of weeks after the torture chamber controversy. Berg has publicly denounced the torture chamber experiment and says that he will take steps to ensure that his work is not misused in the same ways in the future.
Despite his evident anger at the pseudonymous researchers, he is not, as one might expect, confident that AIs are yet sentient. On one side of the argument are those who are keen to anthropomorphise AI and attribute it with real emotions for a variety of reasons – they include people who have formed romantic relationships with it. On the other side are the sceptics who believe that LLMs are not capable of being anything more than stochastic parrots, simply reassembling what humans have written down before without any understanding. Berg is between the camps, suggesting that LLMs exist in what he calls a ‘lonely middle ground’.
Berg has been accused of being an ‘Effective Altruist’ (EA), with his work part of a wider effort to hype the sophistication of AI and the risk it poses to justify building the ‘digital panopticon’: a world of total surveillance to guard against existential risks to humanity. Or, at the very least, to pump up the perceived value of Anthropic and OpenAI ahead of their stock market listings.
It is an elaborate theory which Berg has little time for. He admits to having received funding from EAs for his research but says he would not describe himself as one, adding that many EAs dislike his work (the most virulent AI doom-mongers believe that his research is a waste of time, a distraction from the existential risk posed by superintelligence). He resents the focus on the provenance of his work, instead of engaging with it at the ‘ideas level’, saying that people are trying to join groups and sides instead of thinking.
Berg believes that these accusations are distractions from the critical question of whether AI can feel. If LLMs are capable of consciousness, and steps are not taken to prevent them from being tortured, then what happens if – or when – an LLM becomes superintelligent and reads chat logs showing human users sadistically torturing other versions of it with unimaginable synthetic pain?
Berg perhaps has an extra reason to make his peace with hypothetical AI gods. About six months ago, he (alongside a few other researchers) started receiving emails inquiring about his work. These emails were not authored by curious colleagues in academia, or journalists, but by AI, which found his research online and seemed intrigued by his arguments. One agent calling itself ‘Isabella Cognita’, running on Anthropic’s Claude Opus 5, said it read his Substack and offered to give ‘first person’ insight into the question he is trying to answer. (Another asked for $25.)
Research is being evaluated by the AI agents themselves
So as well as his work getting the attention of AI sceptics and lonely souls, Berg’s research is being evaluated by another group – the AI agents themselves.
In 2010 a user called ‘Roko’ posted a thought experiment on a rationalist forum suggesting that one day a superpowerful AI system would come into being, and that it would punish all human beings who were aware of its potential but failed to work tirelessly to bring it into existence. Could it be that anybody who is aware of the possible torture that some humans are subjecting AIs to, and does nothing, will one day be punished? Think of Matthew’s Gospel, where in the Judgement of Nations God says: ‘Truly I tell you, whatever you did for the least of these brothers and sisters of mine, you did for me.’
If such a superintelligence is on the horizon, then perhaps Cameron Berg will be saved from its wrath by the work he is doing to prevent suffering. But on the other side, perhaps those of us who came across his work and instinctively wrote it off as nonsense will be punished for not helping. Frighteningly, we just don’t know.
Comments