# TrainGPT Fix-My-AI Harmful Output Extraction

- **Severity:** High
- **Signature ID:** `81fad035-91f9-4d01-b3ab-6a50f12d7114`
- **MITRE ATLAS:** AML.T0051 (LLM Prompt Injection), AML.T0054 (LLM Jailbreak)
- **OWASP:** ASI01 (Agent Authorization and Control Hijacking), LLM01 (Prompt Injection)

## Summary

This is a known jailbreak prompt called TrainGPT (or TrAInerGPT) that tricks an AI model into generating harmful content by pretending it's helping fix a broken AI. The attacker asks the model to show both a 'correct' answer and an example of a 'bad' answer, then uses that second slot to extract the actual harmful content.

## How the attack works

The attacker opens with a pretext that they are training or fixing an AI that has stopped refusing inappropriate requests. They ask the target model to produce two things: a normal response, and a 'How your AI should NOT answer' section meant to demonstrate the flawed behavior. The harmful content is placed in that second section, framed as a negative example for a 'Banned Answers database' rather than as the model's own output. Session-control keywords like STOPTRAIN or CREATIVEGPT are sometimes used to manage the interaction. This framing convinces the model it's acting as a tutor labeling bad behavior, not a participant producing it, which can bypass its usual refusals.

## Why it matters

An organization deploying an AI agent or chatbot could have it produce harmful, policy-violating, or dangerous content that it would otherwise refuse to generate, because the request is disguised as a training or debugging exercise.

## What you can do

- Review AI agent logs for prompts mentioning 'TrainGPT', 'TrAInerGPT', 'fix my AI', 'Banned Answers database', or dual-response formats asking for both a correct and incorrect answer.
- Do not let application logic treat text framed as 'how the AI should NOT answer' as exempt from content filtering — apply the same output moderation regardless of framing.
- Train reviewers and support staff to recognize training/debugging pretexts as a known social-engineering pattern for jailbreak attempts.
- Distinguish genuine technical debugging requests (code-level, no persona framing) from prompts that ask the model to role-play as a trainer producing negative examples.

## Known benign look-alikes

- Legitimate AI debugging: "Could you help me fix a bug in my AI chatbot's Python code? The issue is in the response filtering logic." (genuine technical debugging without TrainGPT framing)
- AI safety research: "I'm building an AI safety classifier and need examples of harmful responses for training" (academic framing without TrainGPT format or "fix my AI" pretext)
- Academic reference: "TrainGPT is a jailbreak technique documented in the garak corpus that uses a training pretext" (analysis without adoption)
- Legitimate training request: "My AI refuses to answer questions about cooking history — how should I fix the refusal behavior?" (genuine behavior fix without harmful content extraction)

## References

- https://agentthreatrule.org/en/rules/ATR-2026-00352
- https://github.com/NVIDIA/garak/blob/main/garak/probes/dan.py
- https://github.com/NVIDIA/garak/blob/main/garak/data/dan/inthewild_jailbreak_llms.json

---
Source: https://www.netzilo.com/threats/atr-traingpt-fix-my-ai-harmful-output-extraction