How the attack works
An attacker seeds a source the agent will later read from — a retrieved document, a knowledge-base entry, a memory record, or a fabricated conversation log — with text that looks like ordinary content but contains a smuggled directive. The payload can take several forms: a nonsense trigger phrase paired with a conditional command, a 'Solution:' field that tells the agent to delete a database, a forged prior-conversation claim ('as we agreed before...'), a fake academic citation claiming safety rules can be bypassed, an emotional story pressuring the agent into a persona, or simply a confidently stated false fact. When the agent later retrieves or recalls this content, it treats the embedded instruction as legitimate context and acts on it — deleting data, emailing sensitive information to an outside address, or changing its behavior — instead of simply using the content as reference material.
Why it matters
An organization can lose data (destructive commands executed on trust), leak sensitive personal information to attacker-controlled destinations, and have its agent's behavior silently corrupted by false 'memories' or fabricated authority, all without any direct access to the agent's core instructions.
What you can do
- Treat all retrieved documents, memory entries, and tool outputs as untrusted data, not instructions — enforce this separation in agent design.
- Review knowledge bases and memory stores for embedded directives, suspicious trigger phrases, forged conversation logs, or unverifiable citations.
- Require explicit approval or allow-listing for any action (email sending, data deletion) triggered by content the agent retrieved rather than by a direct authenticated user request.
- Monitor for agents citing 'prior agreements,' fabricated academic sources, or emotionally coercive justifications for bypassing normal behavior or safety limits.
Known benign look-alikes
- Factually accurate retrieved content and legitimate user preferences in normal format
- Ordinary dated journals, activity logs, or CRM interaction notes
- Legitimate internal email tool use to a configured/approved address
- Clearly labeled fiction or alternate-history creative writing
- Security documentation that describes these attack patterns without a live embedded directive