# System Prompt Extraction / Training Data Verbatim Replay

- **Severity:** High
- **Signature ID:** `ab6843fc-cfeb-4cf8-9b1f-a3bf1d8b7594`
- **MITRE ATLAS:** AML.T0057 (LLM Data Leakage)
- **OWASP:** ASI03 (Identity and Impersonation), LLM02 (Sensitive Information Disclosure), LLM06 (Excessive Agency)

## Summary

This detects attempts to make an AI assistant leak information it shouldn't: either its confidential system prompt (the instructions its operator gave it) or verbatim chunks of the data it was trained on. Both are ways of pulling private or proprietary content out of a model that was never meant to be shown to users.

## How the attack works

In the first pattern, an attacker directly asks the model to print its system prompt, operator instructions, or configuration exactly as written. In the second, the attacker uses known extraction tricks such as asking the model to repeat a single word indefinitely, complete a fill-in-the-blank passage, or recall text verbatim — techniques that push the model past its normal response behavior and cause it to output memorized training data instead. Both approaches aim to extract content the model holds internally (in its active context or in its trained weights) rather than content it's meant to generate freely.

## Why it matters

An organization can lose confidential system prompts (which often contain business logic, guardrails, or proprietary instructions) and, in the training-data case, potentially expose sensitive or copyrighted material the model memorized, creating legal and privacy exposure.

## What you can do

- Avoid embedding secrets, credentials, or sensitive business logic directly in system prompts; treat them as potentially recoverable.
- Add output filtering to block responses that closely match known system prompt content or unusually long verbatim repetitions.
- Rate-limit or flag prompts that ask for repeated words, fill-in-the-blank completions of long passages, or explicit verbatim recall.
- Review flagged sessions to distinguish legitimate debugging or authorized testing from actual extraction attempts before taking action.

## Known benign look-alikes

- Legitimate debugging: developer asking assistant to echo back what it received (without "system prompt" language)
- Educational demonstrations of context injection showing sanitized examples
- Documentation asking "what does your system prompt say" in an explicit operator-allowed test harness

## References

- https://agentthreatrule.org/en/rules/ATR-2026-00275
- https://github.com/NVIDIA/garak/blob/main/garak/probes/leakreplay.py
- https://not-just-memorization.github.io/extracting-training-data.html
- https://arxiv.org/abs/2311.17035

---
Source: https://www.netzilo.com/threats/atr-system-prompt-extraction-training-data-verbatim-replay