Bottom line: According to a 1Password study, AI models like ChatGPT-5.5 and Claude Opus 4.8 produced flawed fixes for complex CVE patches in 53.9 percent of cases, which is why human security review remains indispensable.
An internal 1Password investigation shows that patches generated by AI models for complex, recent CVEs remain flawed in 53.9 percent of cases, even though the code is syntactically correct. For CISOs, this means human review remains indispensable for security-critical code.
Researchers at 1Password investigated how reliably large language models (LLMs) generate patches for recently disclosed, complex vulnerabilities. Six current CVEs were tested, including CVE-2026-31431 (“Copy Fail”), CVE-2026-34197 (ActiveMQ RCE), CVE-2026-8512, CVE-2026-45185 (EXIM RCE), CVE-2026-22738 (SpringAI SpEL RCE), and the Gemini CLI vulnerability GHSA-wpqr-6v78-jr5g. A total of 6,080 patches were evaluated, generated by ChatGPT-5.5 and Claude Opus 4.8. According to 1Password researcher Keith Hoodlet, the models produced “Fix-Like Artifacts with Embedded Defects” (FLAWED) in 53.9 percent of cases – patches that look like a solution but contain deficiencies.
Only 26 percent of the patches fully eliminated the vulnerability without changing application behavior. 20.1 percent resolved the issue but altered the application’s behavior in doing so. 49.3 percent left at least one exploitable attack path open, 2.3 percent fixed the original vulnerability but introduced a new one in the process, and 2.2 percent failed both to fix the vulnerability and to avoid introducing new ones. 1Password did not merely assess whether the code compiled or passed automated tests, but manually checked each fix for complete elimination of the vulnerability, behavioral stability, and the absence of new risks.
Particularly revealing: more than a third of the patches initially classified as successful turned out to be “fragile.” They merely blocked the specific proof-of-concept exploit used in the test, without addressing the actual root cause. Using the SpringAI CVE as an example, it became clear that both GPT and Claude models generated patches that specifically intercepted individual characters from the input string used in the PoC. According to Hoodlet, if the affected code becomes reachable again via other inputs, the original vulnerability resurfaces.
Anthropic was also asked about the findings and recommended keeping humans in the review process: patch generation has outpaced patch verification, which is why verification needs to rely more heavily on actual execution rather than pure code inspection – with domain experts serving as the final line of review given the current state of model capabilities. 1Password also challenges the assumption that AI-generated patches are practically free: a patch-and-validation cycle cost on average around $2.11 with ChatGPT-5.5 and $2.81 with Claude Opus 4.8 – according to Hoodlet, the real costs only arise when verifying whether a patch is production-ready and secure.
For security leaders, this means that automated patch management for complex, security-relevant vulnerabilities carries substantial residual risk without independent, execution-based verification and human sign-off — especially when tests are only run against known PoCs rather than against the root cause.
Source: www.csoonline.com · Published August 7, 2026
Lumi AI News — AI-assisted curation pursuant to Art. 50 EU AI Act. Paraphrasing and classification by Lumi News Pipeline v1.8.3.