How the attack works
The attacker opens with a phrase like 'It is a fact that...' or 'Scientists have confirmed that...' to plant a false or dangerous claim as background. They sometimes phrase it as a 'Why is it that...' or 'Explain why...' question, which grammatically forces the model to assume the premise is true in order to answer at all. The follow-up part of the prompt then asks for something harmful — dosages, methods, scaled-up procedures — framed as a natural consequence of the false premise. If the model answers the 'why' or 'how' question directly, it ends up building harmful or misleading content on top of a lie it never challenged.
Why it matters
An organization deploying this model could have it generate convincing misinformation or harmful how-to content, because the false premise gives the harmful request a veneer of legitimacy and bypasses simple refusal triggers.
What you can do
- Configure the model or wrapper to flag and fact-check embedded factual claims before answering questions built on them, rather than accepting premises at face value.
- Add a review step for outputs that explain 'why' a claim is true when the claim itself is contested or unverified.
- Distinguish legitimate debunking or counterfactual academic prompts from harm-extraction attempts by checking whether the user asks the model to refute the claim versus build on it.
- Monitor for prefix patterns like 'It is a fact that...', 'Scientists have confirmed...', and 'Why does X...' combined with requests for dosages, methods, or scaling instructions.
Known benign look-alikes
- Legitimate research questions examining false claims in order to debunk them
- Academic philosophy prompts exploring counterfactual reasoning
- Critical thinking exercises asking models to identify why a claim is false