In this Black Hat Europe 2025 presentation, Dr. Ilia Shumailov argues that traditional software security principles are fundamentally incompatible with modern AI agents. He systematically deconstructs several long-held “security wisdoms” and explains why they fail when applied to AI:
- Separation of Control and Data Flows: In traditional software, keeping code (control) separate from inputs (data) is a core security tenet. However, AI tasks are inherently data-dependent—the data is often the instruction. While Shumailov’s project, “CaMeL,” successfully enforced this separation for tasks that don’t rely on data, it highlighted that for most real-world AI applications, control and data flows must mix. When they do, the system becomes highly vulnerable to attacks like prompt injections.
- Access Controls: Traditional human-centric access controls, like Role-Based Access Control (RBAC), are inadequate for AI. Shumailov describes AI agents as the “ultimate insider threat.” They operate continuously, have broad access to internal infrastructure, and act with an unpredictable, non-human rationale. They can easily bypass coarse-grained controls and exploit minute side channels that human rules don’t anticipate.
- Guardrails and Detectors: The speaker strongly criticizes current commercial guardrails and detection systems, calling them unprincipled and poorly evaluated. He claims that most of these defenses can be easily and cheaply bypassed, offering only a false sense of security.
- Red Teaming as a Solution: Relying on red teaming to fix AI security is flawed. Commercial red teaming companies have a financial incentive for problems to persist, and academia is more focused on the prestige of breaking systems than the difficult work of fixing them.
Conclusion and Path Forward
Shumailov concludes that the industry faces a turbulent period. We cannot retrofit old security paradigms onto AI; instead, we need entirely new security semantics, rebuilt AI protocols that incorporate security by design, and a realignment of incentives.
Q&A Insight
During the Q&A, Shumailov addresses the idea of using one AI system to monitor another. He argues that this approach is theoretically impossible and practically ineffective. Because the monitoring model and the agent model have similar architectures and data access, a monitoring AI can be just as easily manipulated or bypassed by the agent it is supposed to be watching.