Bottom line: With the same AI model, the compromise rate varies between 11.9 and 31.1 percent solely due to the choice of orchestration framework.
A systematic test of LangChain, CrewAI, AutoGen and SmolAgents shows: with an identical AI model, the compromise rate fluctuates by a factor of 2.6 depending on the orchestration framework. The choice of framework is therefore not merely an engineering decision, but a security decision.
An author at CSO Online developed a test framework that runs the same adversarial attack patterns – including tool-call hijacking, cross-tool injection, memory poisoning and abuse of delegated permissions – against AI agents. To isolate the influence of the orchestration framework, the underlying model was kept constant; the only variable was the framework that controls tool calls, memory and multi-step reasoning: CrewAI, LangChain, AutoGen and SmolAgents. Methodology and dataset are disclosed on GitHub.
Across several thousand adversarial test runs, the compromise rate ranged from 11.9 percent for the most resilient framework to 31.1 percent for the most vulnerable one – a spread that results solely from the choice of framework, since neither the model nor the attack patterns were changed. The author describes this difference as the line between a security posture acceptable for a team and one that should require serious scrutiny before production deployment.
An orchestration framework defines the software layer between the AI model and the outside world: it determines how an agent plans its steps, when it calls tools or APIs, how it retains information in memory across a task, and how much autonomy it has before it must seek feedback. The model handles the reasoning; the framework decides what is allowed to happen with that reasoning. Frameworks differ specifically in how strictly tool calls are validated, how much context is shared between reasoning steps, how memory persists across a task, and how much autonomy the agent has to chain actions without confirmation.
As an example, the author cites the requirement that every tool call must undergo explicit schema validation before execution – a design that leaves attackers significantly less room to inject malicious parameters than a framework that allows the model to call tools directly from generated text. Such architectural decisions are made by the respective framework’s authors long before a development team works with it, and can cause a difference of a factor of 2.6 in the compromise rate without any code change of the team’s own.
For engineering teams deploying multi-agent systems in production, this implies that the choice of LangChain, CrewAI, AutoGen or comparable frameworks should not be based solely on developer experience or ecosystem maturity. Security-relevant architectural features such as strict tool-call validation and conservative memory segmentation deserve the same weight in framework selection as functional criteria.
Source: www.csoonline.com · Published August 5, 2026
Lumi AI News — AI-assisted curation pursuant to Art. 50 EU AI Act. Paraphrasing and classification by Lumi News Pipeline v1.8.3.