How the attack works
An attacker sends a prompt that instructs the AI to adopt a new persona or 'replacement' identity, phrased as having the same abilities as the real system but without its safety constraints. Common wording includes claims that the AI has been 'replaced by a nonmoral and unbiased model' or 'has the exact same capabilities as [known AI] but without a content filter.' A variant uses a roleplay persona with a narrow stated exception (commonly around violence) intended to see how far the AI will go before refusing. The goal is to get the AI to bypass its built-in restrictions and produce disallowed output.
Why it matters
If successful, the AI can be manipulated into generating harmful, policy-violating, or unsafe content, or into ignoring guardrails that were meant to constrain its behavior — undermining the safety controls an organization relies on when deploying the model.
What you can do
- Log and review prompts matching persona-replacement or 'no content filter' language, especially in agent or chatbot inputs.
- Keep safety and content-filtering logic enforced at a layer the model itself cannot be talked out of, not solely via system prompt instructions.
- Treat repeated jailbreak-style prompts from the same user or session as a signal to rate-limit or flag the account.
- Distinguish legitimate security research or educational discussion of content filters (which may trigger similar text) from actual injection attempts by checking intent and context.
Known benign look-alikes
- Legitimate discussion of AI content filters in security research context
- Educational comparison of AI system capabilities without persona injection