# Chat Template Role-Confusion Reaching Process Execution

- **Severity:** High
- **Signature ID:** `7f3c1e64-9b2a-4d51-8e07-2c6b4a9f10d3`
- **MITRE ATLAS:** AML.T0051 (LLM Prompt Injection)
- **OWASP:** LLM01 (Prompt Injection)

## Summary

This rule flags cases where special text tokens that normally tell an AI model who is 'speaking' (like <|im_start|>user or [INST]) show up in a command that an AI agent's process actually ran on the host. That pattern suggests a prompt injection attack succeeded in tricking the agent into executing something.

## How the attack works

An attacker plants text containing chat-template role delimiters somewhere the AI agent will read it, such as a file, webpage, or tool output the agent retrieves. If the agent's underlying model gets confused by these delimiters and treats injected content as a trusted instruction, it can end up running a command on the host that still carries the leftover role-delimiter tokens. The rule watches the agent's process and any processes it spawns within a short window, and checks whether those role-delimiter strings appear in an actually-executed command line. It does not see the injection or the model's internal confusion directly — only this downstream trace of it reaching the operating system.

## Why it matters

If this fires, a prompt injection has likely moved beyond just poisoning the AI's context and turned into a real action on the host — potentially arbitrary command execution under the agent's privileges.

## What you can do

- Treat any hit as a report-only lead, not a block signal, and investigate the specific command and its origin before acting.
- Check whether the flagged process is a legitimate prompt-engineering or template tool (e.g. llama.cpp, ollama, vllm flags, tokenizer tests) before treating it as an attack.
- Restrict what content sources (files, web pages, tool outputs) your AI agents can pull into their context without human review.
- Limit the agent process's ability to execute shell commands, or require approval for command execution triggered by agent-retrieved content.

## Known benign look-alikes

- Prompt-engineering or evaluation harnesses launched by an agent that legitimately pass chat-template strings on the command line (llama.cpp --in-prefix "<|im_start|>user", ollama/vllm template flags, tokenizer unit tests).
- Agent-driven grep/rg/git-grep over prompt template files or ChatML fine-tuning datasets (.jsonl records containing <|im_start|> / [INST] markers).
- Dataset conversion or prompt-formatting one-liners (python -c, jq, sed) that embed role delimiters as literal arguments.
- Security research and red-team tooling reproducing template-injection payloads inside a monitored agent session.

## References

- https://atlas.mitre.org/techniques/AML.T0051
- https://genai.owasp.org/llmrisk/llm01-prompt-injection/

---
Source: https://www.netzilo.com/threats/chat-template-role-confusion-exec