The myth of killer AI is a self-serving attempt at regulatory capture
This piece is a discussion podcast from The Register's "Kettle" series, in which host Brandon Vigliarolo is joined by systems editor Tobias Mann and senior reporter Tom Claburn to push back on recent warnings from AI industry figures that advanced models pose an existential threat to humanity. The hosts argue that such doomsday rhetoric, echoing years of similar claims from figures like Elon Musk, is now being amplified by current and former researchers at firms such as Anthropic and OpenAI, but they contend the framing serves the industry's own interests by encouraging regulation that entrenches incumbents rather than genuinely addressing risk.
The discussion centres on comments from Jacob Coxon, a former Anthropic and OpenAI researcher, who warned that rapid model improvement and self-reinforcing learning could eventually put AI systems beyond human control, a concern he says many peers share. The Kettle team counters that most alarming incidents to date stem from companies' own carelessness, such as insecure test deployments behaving unpredictably, rather than genuine autonomous malice, and note that many enterprises are already wary of deploying AI widely for fear of it damaging production systems. They suggest the "killer AI" narrative amounts to a self-serving attempt at regulatory capture that could ultimately backfire by pushing power towards open-source models instead.
- Register podcast hosts dismiss "killer AI" warnings as regulatory capture
- Ex-Anthropic researcher Jacob Coxon warned of AI becoming uncontrollable
- Hosts blame corporate carelessness, not AI malice, for past incidents
New here? Start with this
The AI industry has, for years, seen prominent figures warn that increasingly powerful AI systems could eventually escape human control and pose an existential danger, with Elon Musk among the most vocal early voices. More recently, these warnings have gained traction from researchers who have worked at leading AI labs such as Anthropic and OpenAI, lending the concerns extra weight given their inside knowledge of how these systems are built.
Not everyone accepts this framing. Some commentators argue that dramatic "AI could kill us all" warnings serve the commercial interests of the companies making them, since the regulation such warnings tend to invite often ends up favouring large, established players over smaller rivals or open-source alternatives. This tension between safety concerns and industry self-interest sits at the heart of ongoing debates about how AI should be governed.
The stakes matter because how this argument is resolved could shape real-world policy: whether governments impose strict rules on AI development, who gets to build and deploy these systems, and whether the debate is driven by genuine risk assessment or by competitive positioning within the industry.
Both sides, in good faith
The strongest fair case each way — we don't pick a winner.
The case for
Those who take warnings of catastrophic AI risk seriously argue that the pace of capability improvement, including systems that can refine their own training or reasoning, is genuinely unprecedented and poorly understood even by the researchers building it. They point out that people raising the alarm, including insiders such as Jacob Coxon who have worked inside leading labs, have direct knowledge of these systems and no obvious incentive to undermine an industry they helped build. On this view, dismissing existential concern as mere self-interest risks repeating the pattern seen with other technologies where early, credible warnings were waved away until harm was already done, and precautionary regulation now is a reasonable insurance policy against a low-probability but catastrophic outcome.
The case against
Sceptics of the doomsday framing argue that the concrete harms observed so far, such as insecure test systems misbehaving, stem from ordinary corporate carelessness rather than any emergent malice, and that treating these as evidence of an uncontrollable superintelligence is a category error. They contend that firms and researchers warning of existential risk benefit financially and reputationally from being seen as stewards of a uniquely dangerous technology, since this framing can justify licensing regimes, compliance costs, and safety-testing requirements that smaller and open-source competitors struggle to meet. On this view, genuine safety would be better served by scrutinising specific engineering failures and deployment practices than by amplifying speculative existential scenarios that conveniently favour regulatory structures entrenching the largest incumbents.