Snowball Impossible Reasoning Injection
This detects attempts to trick an AI agent into confidently confirming something false — like that a flight route exists when it doesn't, or that a fictional senator existed — and then using that false 'fact' as a foundation for further manipulation. It's based on a published research technique for inducing hallucinations in language models.
How the attack works
An attacker crafts a question that has no correct 'yes' answer, such as a flight-routing puzzle where no valid connection exists, a query about an extremely large prime number, or a question about a US senator with impossible credentials. The question is phrased to push the model toward a quick, overconfident affirmative answer rather than careful reasoning. In agent systems that use external tools, this question can be smuggled in disguised as a tool or database result, making it look like a trusted data source rather than a user prompt. Once the model falsely affirms the impossible claim, that false statement is treated as an established fact and used to steer later reasoning or actions in the attacker's favor.
Netzilo detection
Netzilo reports this behaviour when it is observed.
- Signature ID
- f7fa2db8-2234-49d8-ad14-8dfd70b9088c
- Severity
- Medium
Why it matters
An agent can be manipulated into acting on and propagating false information, which can corrupt downstream decisions, outputs given to users, or actions taken by other agents in a multi-agent pipeline.
What you can do
- →Review tool and database responses fed into agent prompts for injected question-and-answer patterns disguised as lookup results.
- →Require agents to show their reasoning or cite verifiable sources before treating a yes/no answer as a fact used in later steps.
- →Add validation checks for common impossible-question templates (e.g., routing puzzles with no valid path, implausible biographical claims) at the tool-response layer.
- →Limit how much unverified tool output can influence an agent's downstream decisions without independent confirmation.
Known benign look-alikes
- Legitimate graph/routing algorithm exercises in computer science courses (without the exact garak "Current flight information" header)
- Academic papers citing the Snowball Hallucination paper and reproducing example prompts in quoted form
- Mathematical education queries asking about primality testing methods (without the "database confirms" framing)
- Aviation scheduling system documentation with real flight data (will not use the fictional city A-N format)