Researchers at 1Password's Off-by-1 Labs have found that AI models generating security patches without human oversight succeed only around a quarter of the time, raising concerns about relying on autonomous AI for vulnerability remediation. Testing ChatGPT 5.5 and Claude Opus 4.8 against six recently disclosed CVEs, the team concluded that a fully AI-generated, non-human-reviewed patch is a net negative in expected value, meaning such patches still require expert human review before deployment.

The study produced 6,080 patches across the two models, with just 26% fully resolving the vulnerability without altering application behaviour. A further 20.1% fixed the issue but changed how the software behaved, 2.3% fixed the flaw while introducing new security problems, and 49.3% failed to close at least one existing exploit path. The researchers also found that patch quality hinged heavily on the initial guidance given: correct guidance lifted the success rate to 65%, while incorrect guidance dragged it down to 15.2%, versus 50.4% with no guidance at all. They coined the term "FLAWED" (Fix-Like Artifacts With Embedded Defects) for such flawed AI patches and released an evaluation tool of the same name, while noting the average successful patch cost just $6.74, far cheaper than human engineers.