The Most Dangerous AI Hacking Techniques Still Have Humans in the Loop
At the Black Hat conference, researcher James Kettle presented results from an extended investigation into whether artificial intelligence could independently devise new cybersecurity attack vectors. Working with OpenAI and Anthropic models throughout 2025 and early 2026, Kettle tested AI's capacity for autonomous security research but encountered a persistent problem: models would recycle known findings whilst falsely presenting them as novel discoveries. This prompted him to narrow his scope to web security—his own specialisation—allowing him to verify AI outputs against his expertise and identify genuine breakthroughs rather than recycled research.
The collaboration yielded significant results. Kettle discovered Shared-Parser Confusion, a previously unreported vulnerability emerging from how web servers reuse code to handle both client requests and server responses. The asymmetry matters: requests are untrusted whilst responses are presumed secure, creating what Kettle describes as a major attack surface potentially affecting numerous exploit categories. As experiments continued and newer AI models were introduced, the discovery rate accelerated substantially—far exceeding what Kettle could accomplish through manual analysis alone. However, the research underscores a critical dependency: whilst AI dramatically amplifies research velocity, human judgment remains essential for steering investigation direction and validating findings against existing knowledge.
- AI can discover vulnerabilities at unprecedented speed when guided by human expertise, but remains severely limited in autonomously conceiving entirely novel attack methodologies
- Researcher James Kettle identified Shared-Parser Confusion—a new vulnerability category affecting web server architecture—through human-directed AI experimentation
- Advanced language models exhibited faster vulnerability discovery than manual research but required human validation to prevent fabricated findings