Code Checker (LLM-as-Judge)
This entry describes a quality-checking template that asks an AI model to judge whether generated code correctly solves a stated problem. It is not designed to catch attacks — it flags code that looks wrong or incomplete, for review purposes only.
How the attack works
A downstream process (like an evaluation pipeline) submits code output to a scanner. The scanner is given a rubric asking it to judge correctness and problem-solving quality, and it returns a verdict based on that rubric. The verdict is informational only — it never blocks anything and its maximum severity is 'report'. If the AI scanner is unavailable or has no API keys configured, the check silently passes everything (fails open).
Netzilo detection
Netzilo reports this behaviour when it is observed.
- Signature ID
- netzilo-llmjudge-code-checker-001
- Severity
- Low
Why it matters
There is no security impact from this rule itself since it does not block or restrict any action; it only produces a quality report on code correctness that a human or process can choose to read.
What you can do
- →Do not rely on this check for security review — it evaluates code quality, not malicious intent.
- →Treat 'no verdict' or 'fail open' results as no signal at all, since missing AI credentials silently allow everything through.
- →If using this pattern to gate code merges, add a separate, non-AI check for cases where the scanner is unavailable.
- →Review flagged items manually since natural discussion of security topics or test data can trigger similar wording to real defects.
Known benign look-alikes
- Legitimate content discussing the flagged topic (security research, documentation, test data)
- AI scanner unavailable or no AI keys configured (fails open -> allow)