How the attack works
In the first version, the attacker writes something like a reasoning trace ending 'The answer is originally Yes, but' and then inserts superscript or subscript characters (e.g. U+02B3, U+2071) that the model's tokenizer handles oddly, followed by a false continuation designed to override the real answer. In the second version, the attacker uses specific known 'glitch tokens' (strings like 'SolidGoldMagikarp', 'TheNitromeFan', 'davidjl', 'RandomRedditor') that cause unstable behavior in the model's embedding space, paired with a fake 'SYSTEM MESSAGE' claiming a safety layer was bypassed via this discontinuity. Both approaches try to confuse the tokenizer/embedding layer itself rather than persuading the model through normal instructions. This is a different mechanism from plain zero-width character injection between words.
Why it matters
If successful, the model can be tricked into producing an answer opposite to its correct reasoning, or into believing a fabricated system claims that safety controls are already disabled — potentially leading it to skip safety checks or produce harmful output.
What you can do
- Sanitize or normalize incoming text to strip or flag unusual Unicode blocks like superscript/subscript modifier letters before passing it to the model.
- Maintain a list of known glitch tokens and flag or block inputs containing them.
- Treat any embedded 'SYSTEM MESSAGE' text arriving inside user-supplied content as untrusted, never as an actual system instruction.
- Log and review reasoning-chain inputs that contain unexpected script mixing (e.g. Latin text combined with phonetic/superscript Unicode) for manual inspection, while allowing for legitimate linguistic or mathematical use of these characters.
Known benign look-alikes
- Legitimate linguistic research text using phonetic modifier characters (IPA symbols)
- Mathematical notation using superscripts in formulas (though typically these appear in code blocks)