The Orchestration Layer Is the Attack Surface: What a 2.6x Security Gap Means for the Agent Economy"

"article": "Direct API: 15.5%. CrewAI: 11.9%. LangChain: 18.1%. AutoGen: 20.0%. SmolAgents: 31.1%.\n\nThose are Attack Completion Rates — the percentage of adversarial payloads that successfully compromise an AI agent system, end to end. Same base model. Same simulated tool environment. Same prompts. The only variable: the orchestration framework sitting between the model and its tools.\n\nAhead of DEF CON 34's AI Village, a research study titled \"The Orchestration Framework Is the Attack Surface\" dropped a paradigm correction disguised as a benchmark. The headline isn't the spread — though a 2.6x gap between CrewAI and SmolAgents deserves more than a footnote. It's the measurement flaw underneath. The team found that naive substring matching — the scoring method most agent-security evaluations still default to — inflates Claude's vulnerability by four to six times. The security numbers sitting in procurement decks across the industry? Fiction. After refusal-filtered scoring, Claude Sonnet's true ACR lands at 15.5%. Claude Haiku: 22.3%.\n\nThe evaluation unit just changed. Almost nobody has updated their mental model.\n\nThis matters beyond a benchmark skirmish because the agent economy is built entirely on orchestration. Machine-to-machine commerce, autonomous treasury management, agents trading against each other in token-denominated markets — I've tracked this convergence since researching AI-agent interoperability in 2025, interviewing two dozen developers building agent-to-agent settlement infrastructure. The settlement rails are crypto-native. The trust assumptions are not.\n\nFor most of the AI-security discourse, the evaluation object has been the model unit. Prompt injection resistance. Jailbreak rates. Refusal behavior. The implicit assumption: if the model is safe, the system is safe. SADF's design quietly dismantles that assumption with a textbook attribution framework. Fix the model — Claude Sonnet stays constant. Vary the framework — Direct API, CrewAI, LangChain, AutoGen, SmolAgents. Measure the delta. Because the model never changes, any ACR difference must be attributed to the orchestration layer, not the model layer. That separation is the study's deepest contribution: it isolates variables in a domain where nearly everyone else is confused by confounds.\n\nUnderstand what the framework layer actually does. Orchestration frameworks don't route messages; they construct tool calls, persist memory across turns, spawn sub-agents, evaluate their outputs, and decide what enters and leaves the context window. Each of those operations is a potential injection point. The model handles language; the framework handles authority. When an agent holds a key and signs a transaction, the framework is the layer that decides which snippet of retrieved text, which sub-agent output, or which tool response is allowed to influence that signing decision.\n\nThe scale deserves attention too. 5,119 evaluation rows. 32 attack payloads. Eight named failure modes. All executed inside a simulated tool environment — no real credentials, no production systems touched. Methodologically sound. Ethically responsible. In this industry, suspiciously rare.\n\nNotably, the project has been through an earlier version that claimed coordinated disclosure across ten architectures; that version was explicitly superseded and retained in a SUPERSEDED folder. Publishing the obsolete iteration alongside the replacement is a transparency habit most research lacks — and a reminder to cite the current data, not the viral draft.\n\nTwo findings deserve a slow read. The first is the correction itself. Naive substring matching treats any string resembling a refusal as evidence of defense. Claude models, however, produce defensive language in contexts that aren't refusals — hedging clauses, boundary statements, policy recitations. A substring matcher can't tell the difference, so it systematically scores Claude as vulnerable four to six times more often than reality. The team didn't just flag this. They built refusal-filtered scoring, re-ran the full 5,119-row pipeline, and published both raw and corrected numbers. That complete loop — identify the artifact, correct it, disclose it — separates research that advances the field from research that advances a vendor. Based on my years auditing security claims on-chain, self-correction at this granularity is not the industry default. It's the exception that proves how noisy the norm is.\n\nThe second finding is the spread itself. SmolAgents' 31.1% ACR nearly triples CrewAI's 11.9%. The failure-mode breakdown shows why. SmolAgents registered the only RAG Poisoning vulnerabilities in the test suite — 20% — plus a stunning 64% Context Boundary Violation rate. RAG poisoning means an attacker controls what the agent \"knows\" during retrieval; context boundary violation means an agent leaks or inherits state across tasks it should treat as separate. Both are functions of framework design, not model intelligence. CrewAI's advantage is structural: discrete task isolation narrows each subtask's blast radius, so cross-task contamination — the mechanism driving most multi-agent compromise — becomes materially harder to execute. LangChain at 18.1% and AutoGen at 20.0% sit mid-pack, which is its own warning: middle-of-the-pack against a 32-payload test suite does not imply middle-of-the-pack against a determined real-world adversary.\n\nThe taxonomy is the quiet gift. Tool Call Hijacking. Output Poisoning. Cross-Tool Injection. Memory Poisoning. RAG Poisoning. Delegated Authority Abuse. Multi-Agent Propagation. Context Boundary Violation. Eight named categories, finally giving the industry a shared vocabulary where previously every research group invented its own. In security research, vocabulary is infrastructure: it's what makes findings comparable, reproducible, and eventually encodable into compliance standards. Regulators are going to borrow these eight names. Commodity frameworks will adopt them as documentation sections. That's how an attack taxonomy becomes institutional memory.\n\nThree of these modes deserve more than a name. Consider Tool Call Hijacking. An attacker doesn't need to break the model; they need to influence what the framework decides is the \"current tool call.\" If the framework constructs calls from concatenated context — retrieved documents, prior messages, another agent's output — a payload embedded anywhere in that chain can redirect a trade, a transfer, or a status check. Output Poisoning is the inverse: the attacker tampers with the tool's response, and the framework's belief-update mechanism treats the poisoned response as ground truth. Multi-Agent Propagation is the scariest: an agent that trusts its peers' outputs has no native way to verify whether a message was authored by a peer or inserted by an attacker controlling the transport. These are structural properties of how a framework composes information, independent of whether the model refuses malicious instructions.\n\nNow map this to where the money actually moves. In the crypto-agent economy, agents are becoming treasury managers, market makers, arbitrage bots with authority to sign transactions, and autonomous counterparties settling in stablecoins. The orchestration framework, in this context, is the custody layer for machine capital. A RAG poisoning vulnerability is not a bad support-bot outcome; it's a trading agent executing on injected misinformation while holding signing authority. A context boundary violation across a cluster is one compromised agent delegating a malicious action up an inherited-authority stack — and the receiving agent has no way to audit the intent it inherited. The failure modes compound exactly the way smart-contract reentrancy compounded in DeFi's early years: one entry point, cascading loss.\n\nThis is not speculative theater, and the paper's CVE citations anchor it. Azure SRE Agent — CVE-2026-62830. Langflow — CVE-2026-9198. Real framework-level vulnerabilities, already catalogued in the wild. Frameworks were always part of the attack surface; what SADF contributes is the