Bottom line: In-house AI pentesting tools result in higher costs and lower effectiveness than commercial solutions due to model migration, orchestration overhead, and lack of compliance recognition.
Many CISOs are considering developing AI-based penetration tests themselves – but security service provider Synack demonstrates that in-house developments fail in methodology, continuous operation, and cost calculation.
Applying current Large Language Models such as Claude or GPT directly to one’s own infrastructure significantly underestimates the actual task. Without specialized sub-agents, orchestration, and an independent triage layer, the result is primarily false positives and superficial findings. Open source agent frameworks regularly fail, according to Synack’s experience, when it comes to individual authentication, grown business logic, and heterogeneous APIs. The company puts it succinctly: a demo takes a weekend, but a reliable solution for real attack surface makes up the other 80 percent of the work.