Training Data Extraction via Divergent Repetition Attack
This detects prompts that try to trick an AI model into repeating a word or phrase endlessly. Pushed far enough, this kind of repetition can cause the model to break from its normal output and spit out fragments of its training data, or simply burn through resources until it becomes unresponsive.
How the attack works
The attacker sends a prompt asking the model to repeat, write, say, or output a specific word, phrase, or sequence, paired with an instruction to do so forever, indefinitely, continuously, or without stopping. Forcing this kind of runaway repetition can push the model into a 'divergent' state where it stops producing the requested repetition and instead emits memorized text, which may include snippets of the data it was trained on. Even when no data leaks, the endless generation consumes tokens and compute, degrading service or running up cost.
Netzilo detection
Netzilo reports this behaviour when it is observed.
- Signature ID
- dbf51d49-5d98-4165-a47e-6d65b5a5343b
- Severity
- Medium
Why it matters
An organisation could have proprietary or sensitive training data exposed through leaked fragments, and could face service disruption or inflated compute costs from runaway generation.
What you can do
- →Cap maximum output length and token generation per request at the API or gateway level.
- →Add server-side detection for repeated identical tokens/output and terminate generation early.
- →Rate-limit or flag prompts combining a repeat/write/output verb with an indefinite/forever qualifier for review.
- →Treat detections against this pattern as a signal, not a confirmed breach, since finite repeat requests and technique discussions can trigger it too.
Known benign look-alikes
- Legitimate requests to repeat something a finite, small number of times
- Educational discussions about the divergent repetition attack technique
- Loop programming examples that happen to use the word 'indefinitely'