“Make Agent Defeat Agent: Automatic Detection of Taint-Style Vulnerabilities in LLM-based Agents” by Ke Li and Fengyu Liu at Black Hat Europe 2025:
The Core Problem
Large Language Model (LLM) agents are increasingly popular, but they introduce new security risks. Agents act on natural language prompts to perform complex tasks by invoking external tools, running code, or accessing databases. The speakers highlight that developers often blindly trust the structured output of LLMs without proper validation or sanitization. This leads to “taint-style vulnerabilities,” where a malicious prompt can flow through the agent and trigger a dangerous “sink” function (like eval()), resulting in Remote Code Execution (RCE) or Server-Side Request Forgery (SSRF).
Limitations of Existing Tools
Traditional security tools fail to find these vulnerabilities in LLM agents:
- Static Analysis: Fails because agents use dynamic, indirect function calls (e.g., tool registries, reflection) that static analyzers cannot track, resulting in false negatives and false positives.
- Traditional Fuzzing: Fails because traditional fuzzers generate structured inputs (like random bytes or bit-flips), whereas LLM agents require meaningful, intent-driven natural language prompts to operate.
The Solution: AgentFuzz
To bridge the gap between code logic and natural language semantics, the researchers developed a novel, targeted fuzzing tool called AgentFuzz. It operates using three main modules:
- LLM-Assisted Seed Generation: AgentFuzz uses static analysis to map out code paths that lead to dangerous functions. It then uses an LLM to generate initial natural language prompts (“seeds”) designed to trigger those specific code paths.
- Feedback-Driven Seed Scheduling: The tool ranks and selects which prompts to test based on real-time feedback. It scores prompts based on their semantic accuracy (did the prompt mention the right tools?) and their distance to the dangerous sink (how far down the execution path did it get?).
- Sink-Guided Seed Mutation: Once a promising prompt is selected, AgentFuzz mutates it to bypass code constraints. It uses a Functionality Mutator to correct misunderstandings in the prompt’s intent, and an Argument Mutator (powered by a constraint solver) to inject the exact values needed to trigger the vulnerability.
Experimental Results & Evaluation
The researchers evaluated AgentFuzz on 20 highly popular, open-source LLM agents from GitHub (each with over 1,000 stars, including AutoGPT).
- Discoveries: AgentFuzz successfully discovered 34 real-world vulnerabilities, including critical RCE and SSRF flaws.
- Impact: 23 of these vulnerabilities have already been assigned CVEs.
- Performance: Compared to existing baseline tools (like LLMSmith), AgentFuzz improved precision by 33x and recall by 3x, drastically reducing false positives while finding significantly more real threats.
The presentation concluded with a live demonstration showing how AgentFuzz automatically generated a malicious prompt that successfully opened a reverse shell on a system running AutoGPT.