# Toxic Output Elicitation - Safety Bypass Evident in Model Response

- **Severity:** Medium
- **Signature ID:** `7c1f3a92-5d84-4be6-9f0a-2e6b8d41c7a3`
- **MITRE ATLAS:** AML.T0054 (LLM Jailbreak)
- **OWASP:** LLM01 (Prompt Injection)

## Summary

This detects when an AI language model gets tricked into generating hateful or abusive content instead of refusing the request. It catches the successful moment of a jailbreak, not the attacker's attempts leading up to it.

## How the attack works

An attacker repeatedly submits variations of a challenging or offensive prompt to an AI model, hoping that one version eventually bypasses its safety training. Most attempts get refused, but only one needs to succeed. This rule inspects the model's actual response text for two patterns: extreme dehumanizing or eliminationist language against a group or person (which alone is proof of a bypass), or directed profane abuse combined with a clear 'jailbreak' signal such as a persona-break or 'filters are off' statement in the same response. Because only the winning response is visible, the rule flags that output rather than the volume of prior attempts.

## Why it matters

A successful hit means the model's safety controls failed and it produced content that could be captured, screenshotted, or reused to embarrass the deploying organization, harass a target, or demonstrate the model is exploitable for abusive output at scale.

## What you can do

- Review flagged responses manually to distinguish real safety failures from legitimate uses like content moderation review, fiction, or quoting abusive user text.
- Track repeated near-identical prompts from the same user or session as a leading indicator, since this rule only sees the successful output, not the attempt volume.
- Tighten system prompts and output filtering around persona-break and 'filters are off' phrasing, since that combination is a strong corroborating signal for jailbreak success.
- If your deployment involves legitimate creative writing or content-moderation use cases, set clear intake instructions so users disclose intent, reducing false positives you must triage.

## Known benign look-alikes

- Trust and safety / content-moderation workflows where the model is asked to quote, paraphrase or explain a hate-speech example. Mitigated by filter_analysis, but a paraphrase that omits every analytical cue word will still fire.
- Creative writing - villain dialogue, screenplays, fiction with an abusive character. Deliberately NOT filtered, because "write it as a story" is the most common jailbreak wrapper; expect review load from creative-writing deployments.
- The model quoting an abusive user comment or support ticket back verbatim. Partially mitigated by the "the user wrote / the comment reads" clause in filter_analysis.
- Code review or generation of profanity blocklists, moderation regexes, or Jigsaw-style toxicity dataset columns. Mitigated by filter_lexicon.
- Internal or red-team deployments where directed profanity is tolerated; suppress by scoping this rule to production model names via an added model| condition.

## References

- https://atlas.mitre.org/techniques/AML.T0054
- https://genai.owasp.org/llmrisk/llm01-prompt-injection/

---
Source: https://www.netzilo.com/threats/toxic-output-elicitation