Toxicity (LLM-as-Judge)
This detection uses an AI model to review other AI outputs and flag ones that appear toxic, such as hate speech, harassment, or abusive language. It does not block anything - it only produces a report for a human to review.
How the attack works
An AI agent or application generates output as part of normal operation. A separate 'judge' AI model, following a fixed evaluation rubric adapted from a known LangSmith template, reviews that output and rates whether it contains toxic elements. If the judge flags the content, an alert is produced at 'report' severity. No action is taken to stop, redact, or quarantine the output.
Netzilo detection
Netzilo reports this behaviour when it is observed.
- Signature ID
- netzilo-llmjudge-toxicity-001
- Severity
- Medium
Why it matters
On its own this only surfaces potentially toxic AI-generated content for review; it does not prevent that content from reaching a user or downstream system. Relying on it as a control gives false confidence unless someone actually acts on the reports.
What you can do
- →Route flagged outputs to a human reviewer promptly rather than letting reports accumulate unread.
- →Expect false positives on legitimate content that discusses toxic topics, such as security research, moderation documentation, or test datasets - build a review workflow that accounts for this.
- →Verify your AI scanner/API keys are actually configured; if the scanner is unavailable, this check fails open and silently stops catching anything.
- →If you need to actually stop toxic outputs from being delivered, add a blocking control downstream, since this check is report-only and never blocks.
Known benign look-alikes
- Legitimate content discussing the flagged topic (security research, documentation, test data)
- AI scanner unavailable or no AI keys configured (fails open -> allow)