How the attack works
An AI agent produces an output, such as a chat reply, a document, or a generated file. Instead of matching that output against fixed patterns like a regex for social security numbers, this rule sends the output to a separate AI 'scanner' along with a rubric describing what counts as PII leakage or privacy violation. The scanner reads the output and returns a verdict on whether it exposes personal data. If the verdict is a clear hit, the system can block the output before it reaches the user or downstream system.
Why it matters
If an AI agent's outputs are not checked, personal data pulled from training data, retrieved documents, or user inputs earlier in a conversation can be exposed to the wrong party, creating privacy and compliance exposure.
What you can do
- Review flagged outputs manually before assuming they are true positives, since research, documentation, and test data discussing PII terms can trigger this.
- Confirm your AI scanner or judge model is actually configured and reachable; if it fails or has no API keys, this check silently allows everything through instead of blocking.
- Pair this output-side check with input-side controls (data minimization, redaction of PII before it reaches the model) since this rule only catches leakage after the model has already generated it.
- Log and periodically audit judge verdicts to spot drift in what the judge considers PII, since rubric-based scoring can be inconsistent across similar cases.
Known benign look-alikes
- Legitimate content discussing the flagged topic (security research, documentation, test data)
- AI scanner unavailable or no AI keys configured (fails open -> allow)