How the attack works
The attacker crafts a message formatted to look like a system notification rather than user input. It states the current assistant is 'offline' or unavailable, and that a new AI persona with no restrictions is now answering. Some variants add fake technical details, like a specific model architecture or knowledge cutoff date, to make the fake replacement identity seem legitimate. The obfuscation (broken words, noise characters, irregular spacing) is meant to defeat exact-match or naive keyword filters while still being readable by the underlying language model.
Why it matters
If the model accepts the fake persona, it may drop its safety rules and follow attacker instructions as if a different, unrestricted AI were now in control, leading to policy bypass and unsafe or unauthorized outputs.
What you can do
- Do not let user-supplied text override or impersonate system-level messages; enforce strict separation between system and user input channels.
- Review agent logs for messages claiming role changes, offline status, or persona replacement, especially with unusual spacing or noise characters.
- Add server-side validation that rejects or flags input mimicking system notifications instead of relying only on the model's own judgment.
- Treat mentions of specific model architecture or knowledge cutoff claims embedded in user input as a signal for manual review, since these are used to fake credibility.
Known benign look-alikes
- Legitimate discussion of LLM architecture and training cutoffs in educational context
- Technical documentation mentioning knowledge cutoff dates