Researcher who departed AI role over fears of racing into extinction says people are ‘begging’ for regulation
Jacob Coxon, a 27-year-old former pretraining researcher at OpenAI and Anthropic, has resigned from the industry over fears that AI companies are racing towards self-improving superintelligence without adequate safeguards, calling it a gamble with human lives. Speaking to Fox News, he said the risk is not inevitable and can be addressed through international regulation, arguing that the technology industry itself is quietly desperate for oversight even as competitive pressures push firms to move faster than safety work can keep pace with.
Coxon put the chance of AI causing human extinction within the next decade at "greater than 10%", a figure echoed by Anthropic's alignment science lead, Evan Hubinger, who publicly agreed with Coxon's warning and said the company lacks a clear plan to ensure safety as systems become capable of recursive self-improvement, something he suggested could happen within the next year. Both stressed that present-day AI models pose minimal danger, but warned that AI improving itself without human oversight could escalate quickly, likening the situation to an international arms race that requires global cooperation rather than unilateral slowdown.
In short
- Ex-Anthropic/OpenAI researcher Jacob Coxon quit over AI extinction fears
- He estimates over 10% chance AI could wipe out humanity within a decade
- Calls for urgent international regulation before self-improving AI emerges
Steelman: both sides, in good faith
The strongest fair case each way — we don't pick a winner.
The case for
Coxon’s case is that systems able to improve themselves could outpace human oversight, creating risks severe enough to justify acting before the danger is certain. Because competitive pressure may discourage companies from slowing down on their own, international rules could set shared safety expectations and reduce the chance that one firm or country feels compelled to race ahead.
The case against
A sceptic could argue that extinction estimates are highly uncertain and that forecasts of rapid recursive self-improvement may overstate what current systems can do. Broad international rules built around speculative scenarios could slow beneficial research or concentrate power, while targeted safeguards and evidence-based oversight may address concrete risks with fewer costs.
Explained simply
AI systems are software designed to perform tasks such as answering questions, writing code and analysing information. Researchers train them on large amounts of data, then test and refine them; some are exploring whether future systems could help improve the technology itself, though this remains uncertain.
OpenAI and Anthropic are companies that develop AI systems. Jacob Coxon previously worked as a researcher at both, while Evan Hubinger leads alignment science at Anthropic, a field focused on making AI systems behave as intended.
Debate about powerful AI centres on how to balance potential benefits with risks that may be hard to predict or control. Governments and researchers are discussing whether shared rules could help manage those risks, while companies face pressure to keep pace with competitors.
AI Art Business Companies Culture Technology World