The data shows a 2.6x gap in security posture between two AI agent orchestration frameworks. CrewAI records an Attack Completion Rate (ACR) of 11.9%. SmolAgents hits 31.1%. The model is identical. The difference is the framework. This is not a simulation. This is from SADF research presented at DEF CON 34 AI Village, and it signals a structural shift in how we must evaluate agent security — especially for the growing number of on-chain agents executing transactions, managing liquidity, and automating DeFi strategies.

Follow the gas, not the gossip. The ledger remembers everything. And the ledger here is the ACR metric: 5,119 evaluation rows across 32 payloads, with refusal-filtered scoring corrections. The research fixes Claude Sonnet as the base model, then layers five orchestration targets: Direct API (15.5% ACR), CrewAI (11.9%), LangChain (18.1%), AutoGen (20.0%), SmolAgents (31.1%). The control group — Direct API — establishes the model's baseline vulnerability. Every percentage point above that is the framework's contribution. The conclusion is inescapable: the orchestration framework is the attack surface, not the model.

Context: The Rise of On-Chain Agents

Over the past 18 months, autonomous agents have migrated from Discord bots to on-chain execution layers. Projects like Autonolas, MyShell, and even experimental DeFi agents on Uniswap v4 hooks now hold wallets, sign transactions, and manage pools. The promise is reduced latency and conditional execution. The risk is that the orchestration layer — the code that decides which action to take next — becomes a vector for injection, poisoning, and authority abuse.

Traditional security assessments in crypto focus on smart contract logic: reentrancy, integer overflow, access control. That is the model layer. But an agent's orchestration framework is the equivalent of a smart contract's proxy pattern or upgrade mechanism. It governs how the model interacts with tools, memory, and external data. My experience auditing ERC-20 tokens in 2017 taught me that the most critical vulnerabilities often hide in the plumbing between components — the transfer logic, the approve-and-transferFrom pattern. The SADF research confirms that the same principle applies to AI agents: the framework is where the attacks land.

Core: The On-Chain Evidence Chain

Let me translate the SADF findings into on-chain analogies. The research defines eight failure modes: Tool Call Hijacking, Output Poisoning, Cross-Tool Injection, Memory Poisoning, RAG Poisoning, Delegated Authority Abuse, Multi-Agent Propagation, and Context Boundary Violation. Each maps to a real-world on-chain threat.

• Tool Call Hijacking is akin to a compromised oracle that returns a manipulated price feed. The model calls a tool, but the framework intercepts the call and redirects it to a malicious endpoint. ACR across frameworks for this failure mode ranges from 8% to 25%. In a DeFi agent, that could mean a swap executed on a fake pool.
• Delegated Authority Abuse mirrors a smart contract with overly permissive delegation. If an agent has a wallet with approval to spend user funds, a framework vulnerability could allow an attacker to trigger that delegation without the model's intent. The SADF data shows this failure mode is present in all frameworks, with SmolAgents hitting 64% on Context Boundary Violation — a related issue where the agent's context window leaks between tasks.
• Memory Poisoning is the equivalent of a state manipulation attack on a smart contract. If an agent's memory store is corrupted, subsequent decisions become poisoned. The research found that RAG Poisoning (retrieval-augmented generation) was unique to SmolAgents at 20% ACR. For an on-chain agent that reads from a knowledge base of protocol parameters, that could mean acting on stale or malicious data.

The ledger remembers everything. The SADF methodology is notable for its self-correction. The researchers discovered that simple substring matching overestimates Claude's ACR by 4-6x. They introduced refusal-filtered scoring, which discounts responses that the model refused to answer. The corrected ACR for Claude Sonnet dropped from an inflated range to 15.5%. This is a critical lesson for on-chain security analytics: we must always audit the measurement tool, not just the target. When I built the Bitcoin ETF flow dashboard in 2024, I found that naive inflow/outflow calculations misrepresented institutional behavior because they failed to account for ETF creation and redemption mechanics. The data must be filtered for the right signal.

From the 5,119 evaluation rows, the research provides a taxonomic vocabulary: eight failure modes that can be used to write audit checklists for agent systems. This is analogous to the SWC (Smart Contract Weakness Classification) registry. The industry needs a shared language to discuss agent vulnerabilities, and SADF has delivered the first draft.

Contrarian: Correlation ≠ Causation — Model Safety Is Not System Safety

A common narrative in the AI security space is that a more capable model will naturally be safer because it can better refuse harmful instructions. The SADF data directly contradicts this. The same Claude Sonnet model, when plugged into different frameworks, exhibits ACR ranging from 11.9% to 31.1%. The model's refusal capability is constant. The framework's security posture is the variable.

Data > Narrative. The implication is profound: enterprises cannot simply "buy a safer model" and call it done. They must audit the entire stack — model, framework, tool environment, and memory layer. In the crypto world, this is analogous to the mistake of assuming a secure consensus layer means the entire application is secure. The DAO hack in 2016 was not a consensus failure; it was a smart contract logic failure. Similarly, an agent framework with a secure model but a vulnerable orchestration layer is an accident waiting to happen.

However, the contrarian angle goes deeper. The SADF research, while rigorous, has limitations. The 32 payloads were selected by the researchers. They may not represent the full distribution of real-world attacker payloads. Rare but high-impact paths — such as a multi-step social engineering attack that requires contextual awareness — may be undersampled. Furthermore, the experiments were conducted in a simulated tool environment. Real-world tool permissions, timing, and API responses can introduce additional variance. The ACR values are a lower bound in a controlled setting; the actual attack surface in production could be larger.

Another blind spot: the model is fixed at Claude Sonnet. The interaction effect between model and framework is unknown. Would the ACR ranking change if the model were GPT-5.4 or DeepSeek? The SADF research does not answer this. In my experience modeling Curve Finance liquidity in 2020, I found that parameter sensitivity can invert rankings under different conditions. The same may be true here. Until the research is replicated across multiple models, the framework security rankings should be treated as indicative, not absolute.

Follow the gas, not the gossip. The gossip is that framework A is "safer" than framework B. The gas is the quantitative evidence: 5,119 rows, 8 failure modes, refusal-filtered correction. But even the gas can be misleading if the measurement is not calibrated for the full context.

Takeaway: The Next Week Signal

The SADF research is a POC, not a standardized audit suite. But it sets a clear direction: the next wave of security audits for on-chain agents will focus on the orchestration layer. I expect to see the following signals in the next 7-14 days:

• Framework providers (CrewAI, LangChain, AutoGen, SmolAgents) will issue security advisories or patches addressing the specific failure modes identified. CrewAI's low ACR (11.9%) is a competitive advantage; they may double down on discrete task isolation. SmolAgents will need to address Context Boundary Violation (64%) and RAG Poisoning (20%).
• Security firms like OpenZeppelin or Trail of Bits will begin offering "Agent Framework Security Assessments" as a service, using the SADF methodology as a baseline. The commercialization path is clear: transform the research into a repeatable audit product.
• On-chain agent projects will start publishing ACR scores in their documentation, similar to how DeFi projects publish audit reports. The market will demand transparency. The ledger remembers everything.

The data shows that the framework is the attack surface. The next step is to make that measurement a standard part of every agent deployment — on-chain or off-chain. The question is not whether your model is safe. The question is whether your orchestration framework can withstand a payload designed to break it. The answer, as of DEF CON 34, is that it depends on which framework you chose. And now you have the data to decide.

Based on my experience auditing smart contracts in 2017 and building institutional flow dashboards in 2024, I see a clear parallel: the most dangerous vulnerabilities are not in the core logic, but in the integration layer. The SADF research has done the blockchain security community a service by extending this principle to AI agents. The work is not complete — but it is a necessary beginning.