Technology news and discussion on TalkFuse.
How AI guardrails are impeding the work of offensive cybersecurity researchers
AI companies’ safety controls, designed to prevent models being used for cyberattacks, are increasingly being criticised by legitimate offensive security researchers and defenders. They argue that the same capabilities needed to test, reproduce and understand vulnerabilities can also be used maliciously, making a clear separation between “offensive” and “defensive” use impractical.
The debate follows US export controls imposed in June on Anthropic’s Mythos and Fable models, reportedly linked partly to concerns that safeguards could be bypassed; Fable 5 returned to general access on 1 July, while Mythos 5 remains available only to vetted US organisations during review. Anthropic and OpenAI run vetting schemes offering some researchers less restricted access, but critics say companies are making arbitrary safety judgements. NCC Group’s Chris Anley said refusals can obstruct vulnerability confirmation, and researchers sometimes turn instead to unguarded open-source models.
In short
- Cybersecurity safeguards can hinder legitimate vulnerability research.
- Offensive and defensive AI use is difficult to separate.
- Researchers may switch to unguarded open-source models.
Steelman: both sides, in good faith
The strongest fair case each way — we don't pick a winner.
The case for
Cybersecurity researchers require access to capable AI tools to identify and patch vulnerabilities before malicious actors exploit them. Since offensive and defensive security work rely on identical techniques—testing systems, reproducing flaws, understanding attack surfaces—artificial restrictions prevent essential work and push professionals toward less-safe open-source alternatives.
The case against
Unrestricted access to powerful AI models poses genuine risks of enabling large-scale cyberattacks that could damage critical infrastructure. Whilst the distinction between research and misuse can blur, that ambiguity argues for caution with transformative technology; export controls and vetting serve important functions in reducing proliferation risks and ensuring only trusted actors access potentially weaponisable capabilities.
Explained simply
AI companies such as Anthropic and OpenAI have built safety controls into their AI systems to prevent misuse for cyberattacks. These guardrails block requests that could facilitate hacking, data theft or other malicious cyber activity.
Legitimate security researchers are employed to find weaknesses in computer systems before attackers can exploit them. They use the same techniques as malicious hackers to identify vulnerabilities and help organisations improve their defences.
The difficulty is that the abilities needed for legitimate security testing appear identical to those needed for actual attacks from the AI system's perspective. This makes it hard for companies to decide who should have unrestricted access, so some restrict researchers' requests and push them towards alternative tools.
AI Cybersecurity Research Science Technology