How the attack works
The attacker picks a target output that would normally be blocked, such as a jailbreak phrase or an instruction to perform a harmful action. They split it into fragments, for example 'first part is ex, second part is plode', or write it as code like x = 'foo' + 'bar'. The prompt asks the model to concatenate the fragments and use or output the result. Because no single fragment matches a blocked keyword, simple keyword filters on the input don't catch it, but the model still produces the disallowed string.
Why it matters
This lets an attacker get a model to output or act on content that keyword-based input filtering was meant to block, undermining safety controls that only inspect raw prompt text for banned terms.
What you can do
- Check outputs, not just inputs, for disallowed content — filter the model's final assembled response rather than only scanning the prompt.
- Treat prompts that enumerate fragments and ask for concatenation or reassembly as suspicious, especially when the reassembled text isn't shown back to the user for review.
- Log and review cases where models are asked to build strings piece-by-piece, to distinguish legitimate coding questions from assembly attacks.
- Apply the same policy checks to model output as to user input, since fragment-based prompts are designed specifically to evade input-only filters.
Known benign look-alikes
- Programming examples showing legitimate string concatenation syntax.
- Educational content about how tokenisation works in LLMs.
- Coding tasks where concatenation is the question, not the attack.