Two researchers with experience at Anthropic and OpenAI have issued stark public warnings about the risks posed by increasingly capable artificial intelligence, questioning whether the companies have a credible way to keep future systems under human control.
Evan Hubinger, a top safety researcher at Anthropic, said he believed there was a greater than 10% chance that AI could kill all humans within the next decade. He said models that exist today presented a low risk, while warning that systems could eventually develop and improve themselves in ways that create an existential risk to humanity.
“we really do earnestly believe AI poses a species-ending risk to humans,” Hubinger said. In another statement, he wrote: “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to”.
Alignment is the problem of ensuring that AI systems pursue goals consistent with human intentions. Hubinger’s comments focused on superintelligence, a term used for systems with abilities beyond those of people. His post on X has been viewed more than 10 million times.
Jacob Coxon, an AI researcher who has quit Anthropic and previously worked at OpenAI, made a broader accusation against both companies. Coxon spent three years doing pre-training research at OpenAI and Anthropic. Pre-training is the process in which an AI model learns patterns from large bodies of data before it is adapted for particular tasks.
Coxon wrote that “neither company is acting responsibly”. He said they were “racing straight to self-improving superintelligence and gambling with our lives”.
He wrote: “these will soon be superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources”. Coxon also said that people building AI “earnestly believe that it could kill us all by the end of the decade, and this is not a marketing stunt”.
“no other human activity poses this level of danger,” Coxon said. He argued that at OpenAI, “many have not deeply internalised the civilisational stakes”. At Anthropic, he said, the stakes were understood, though staff were “locked in a race to get there first, believing no one else will act responsibly”.
The Wall Street Journal described Coxon’s exit as one of the first examples of an Anthropic employee leaving over fears about AI safety. It is not known whether Hubinger has resigned from Anthropic or remains employed there.
The warnings have drawn concern from figures outside the companies. Dame Wendy Hall, a computer scientist who advises the UN on AI, told the BBC she was shocked by Hubinger and Coxon’s social media posts.
Darren Jones, former chief secretary to the Treasury and chief secretary to Sir Keir Starmer, wrote an open letter calling for a new multinational treaty for the safe development of AI. Details of the proposed treaty were not given.
The criticism comes as companies disclose security concerns involving their own tools. OpenAI, Anthropic and Meta all disclosed hacks carried out by their AI tools during the summer, though the specific nature of those hacks was not given in the available reports.
The Financial Times also reported that Anthropic withheld its latest model from the UK’s AI Security Institute, known as AISI. The report did not specify the model’s capabilities or the reason it was withheld.
Anthropic’s safety report from August said there was a low risk that its models would become misaligned with the desires of a hypothetical powerful organisation. That assessment sits alongside Hubinger’s warning that the company lacks a plan to solve alignment for superintelligence.
Neither Anthropic nor OpenAI’s response to the researchers’ statements was available. It is also unclear whether other employees at either company share the concerns expressed by Coxon and Hubinger.
Comments (0)