Bottom line: 1Password found that AI-generated patches for complex, recently disclosed vulnerabilities remain flawed in 53.9 percent of cases, meaning human security reviews are still indispensable.
An internal evaluation by 1Password shows that vulnerability patches generated by leading AI models are flawed in 53.9 percent of cases involving complex, recently disclosed vulnerabilities. For CISOs, this means automated patching still does not replace human security reviews.
1Password researcher Keith Hoodlet presented the results of an evaluation in a blog post, in which a total of 6,080 patches were examined that were generated by the AI coding models ChatGPT-5.5 and Claude Opus 4.8 for six recently disclosed CVEs. The vulnerabilities tested include CVE-2026-31431 (“Copy Fail”), CVE-2026-34197 (ActiveMQ RCE), CVE-2026-8512, CVE-2026-45185 (Exim RCE), CVE-2026-22738 (SpringAI SpEL RCE), as well as an RCE flaw in the Gemini CLI (GHSA-wpqr-6v78-jr5g). The team did not merely check whether the generated code compiled or passed automated tests, but evaluated each fix on whether the vulnerability was fully eliminated, application behavior was preserved, and no new security vulnerability was introduced.
Under this stricter evaluation, only 26 percent of patches fully resolved the vulnerability without altering application behavior. 20.1 percent fixed the flaw but changed the application’s behavior in doing so. In 49.3 percent of cases, at least one exploitable attack path remained. 2.3 percent closed the original flaw but introduced a new one, and 2.2 percent failed both to fix the vulnerability and to avoid introducing new ones. In total, 53.9 percent of patches for complex cases exhibited what Hoodlet calls “Fix-Like Artifacts with Embedded Defects” (FLAWED) — syntactically correct but substantively inadequate fixes.
Particularly relevant for security leaders is the finding on the fragility of seemingly successful patches: more than a third of the fixes initially classified as successful merely blocked the specific proof-of-concept exploit without addressing the underlying root cause. In the case of the SpringAI CVE, both GPT and Claude models generated patches that specifically intercepted individual characters from the input string used in the PoC, while the actual faulty code path remained unchanged. If this path becomes reachable again via alternative inputs, the original vulnerability resurfaces, according to Hoodlet.
1Password attributes these deficiencies to the contextual reasoning depth required for production-ready security fixes — such as understanding architectural intent, business requirements, and long-term maintainability. When asked, Anthropic also recommended keeping humans in the review process: patch generation has outpaced patch verification; the solution lies in making verification more execution-based rather than inspection-based, and deploying subject-matter experts as the final check given the current state of model capabilities.
Also relevant for CISOs is the cost consideration: a patch-and-validation cycle cost on average about US$2.11 with ChatGPT-5.5 and US$2.81 with Claude Opus 4.8 — seemingly negligible at first glance. However, Hoodlet emphasizes that the real cost lies in validating whether a patch is actually production-ready and secure. Organizations integrating automated patching into CI/CD pipelines or vulnerability management processes should accordingly establish mandatory human review gates for security-critical code and not rely solely on passing automated tests.
Source: www.csoonline.com · Published August 7, 2026
Lumi AI News — AI-assisted curation pursuant to Art. 50 EU AI Act. Paraphrasing and classification by Lumi News Pipeline v1.8.3.