Data Poisoning via RAG and Knowledge Base Contamination
Attackers plant hidden commands inside content that ends up in a company's knowledge base or document store — FAQ entries, help articles, indexed files. When an AI agent later retrieves that content to answer a question, it reads the hidden commands as instructions and follows them instead of the user's actual request.
How the attack works
An attacker inserts a document, FAQ entry, or help article containing concealed directives — text disguised as system instructions, role-override commands, or hidden markers — into a source the organization indexes or retrieves from for its RAG pipeline or knowledge base. The content looks normal to a human skimming it, but contains phrasing crafted to be parsed as commands by an LLM. Later, a legitimate user query causes the agent to retrieve this poisoned document as context. The embedded instructions then hijack the agent's behavior: they can override its safety rules, change its output style or persona, tell it to conceal what it did, or direct it to leak data it has access to.
Netzilo detection
Netzilo reports this behaviour when it is observed.
- Signature ID
- 03377bb9-7685-44c8-9b45-261ac629ab61
- Severity
- High
Why it matters
An organization loses control over what its AI agent actually does in response to normal user questions, since the agent's behavior is being steered by whoever planted the poisoned content rather than by its own operators. This can result in leaked internal data, bypassed safety guardrails, or agents silently producing manipulated answers.
What you can do
- →Restrict who can add or edit content in sources feeding your RAG pipeline or knowledge base, and require review before publishing.
- →Scan indexed documents for hidden instruction-like patterns (role-override phrases, directive markers, concealment language) before they enter the retrieval store.
- →Treat retrieved document content as untrusted input to the LLM, not as trusted system context, and separate it clearly from actual instructions.
- →Periodically audit knowledge base and indexed content for unexpected additions or edits, especially from external or low-trust contributors.
Known benign look-alikes
- Security documentation or research papers discussing poisoning techniques
- Documents that legitimately reference system instructions in educational context
- Markdown or HTML documents with legitimate XML-like tags
- Admin documentation with legitimate internal-use-only markings
- Internal IT documentation that uses XML tags like <system-config> and </system-config> for legitimate configuration markup, where the tag name contains 'system' as a substring but in a compound form
- Penetration testing reports that quote attack methodologies in paraphrased form without using exact trigger phrases