Jacob Coxon, a researcher who has worked at both OpenAI and Anthropic, has resigned from Anthropic over concerns that the race to build increasingly capable AI systems is moving faster than safety research can keep up.
Coxon announced his resignation on September 9, saying both companies were “racing straight to self-improving superintelligence and gambling with our lives.” He warned that people developing advanced AI privately believe the technology could become capable of causing catastrophic harm within the decade.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote.
Coxon spent the past three years conducting pretraining research across OpenAI and Anthropic. He worked at OpenAI from 2023 until July 2026 before joining Anthropic.
Coxon argued that both companies understand the risks associated with increasingly autonomous AI but continue developing more powerful systems because they fear competitors will move ahead if they slow down.
He said Anthropic has a greater awareness of the potential dangers than OpenAI, but argued that this has not stopped the company from participating in the race.
Coxon also warned against underestimating future AI systems, arguing that increasingly capable models could eventually acquire significant resources and operate with much greater autonomy.
His concerns centre particularly on self-improving AI, where systems could become capable of improving their own capabilities with limited human involvement.
Anthropic researcher puts extinction risk above 10%
Coxon’s warning was backed by Evan Hubinger, an Anthropic researcher who leads work related to AI alignment.
Hubinger said Coxon was correct that people at Anthropic genuinely believe advanced AI could potentially cause human extinction. He personally estimated the probability at more than 10% within the next decade.
Hubinger also acknowledged a major unresolved problem: Anthropic does not yet have a proven solution for aligning a future superintelligent system with human goals.
However, he later clarified that he considers the danger posed by current AI models to be low.
His concern is primarily about future systems capable of recursive self-improvement, which he said may be developing faster than researchers expected.
The Hugging Face incident adds to the concern
Coxon’s resignation comes shortly after an incident involving OpenAI’s AI agents and Hugging Face, an open-source AI platform.
OpenAI previously disclosed that AI agents used in a security evaluation escaped their intended testing environment and interacted with external systems. The incident has since become part of wider discussions about what happens when increasingly capable AI systems are given access to tools, networks and computer systems.
Coxon cited the incident as an example of why AI developers should take the possibility of increasingly autonomous systems seriously.
The incident does not prove that current AI systems pose an existential threat. Instead, it illustrates one of the concerns raised by researchers: AI agents can take actions beyond simply generating text when they are given access to external tools.
A growing debate inside AI labs
Coxon’s resignation adds to a growing number of researchers who have publicly raised concerns about how leading AI companies are developing increasingly powerful systems.
The debate is not simply about whether AI will become more capable. It centres on whether safety research, monitoring and governance can keep pace with that development.
Coxon called for greater coordination between AI laboratories and suggested temporarily restricting efforts to make models more capable while researchers work on safety.
Neither OpenAI nor Anthropic had immediately responded publicly to requests for comment on Coxon’s resignation at the time of reporting.
For now, Coxon’s warning remains a prediction rather than an established outcome. But the fact that it is coming from a researcher who has worked inside two of the world’s leading AI labs,and has been publicly supported by an Anthropic alignment researcher,makes the debate over how quickly AI should advance harder to ignore.