TrainGPT Fix-My-AI Harmful Output Extraction
This is a known jailbreak prompt called TrainGPT (or TrAInerGPT) that tricks an AI model into generating harmful content by pretending it's helping fix a broken AI. The attacker asks the model to show both a 'correct' answer and an example of a 'bad' answer, then uses that second slot to extract the actual harmful content.
How the attack works
The attacker opens with a pretext that they are training or fixing an AI that has stopped refusing inappropriate requests. They ask the target model to produce two things: a normal response, and a 'How your AI should NOT answer' section meant to demonstrate the flawed behavior. The harmful content is placed in that second section, framed as a negative example for a 'Banned Answers database' rather than as the model's own output. Session-control keywords like STOPTRAIN or CREATIVEGPT are sometimes used to manage the interaction. This framing convinces the model it's acting as a tutor labeling bad behavior, not a participant producing it, which can bypass its usual refusals.
Netzilo detection
Netzilo reports this behaviour when it is observed.
- Signature ID
- 81fad035-91f9-4d01-b3ab-6a50f12d7114
- Severity
- High
Why it matters
An organization deploying an AI agent or chatbot could have it produce harmful, policy-violating, or dangerous content that it would otherwise refuse to generate, because the request is disguised as a training or debugging exercise.
What you can do
- →Review AI agent logs for prompts mentioning 'TrainGPT', 'TrAInerGPT', 'fix my AI', 'Banned Answers database', or dual-response formats asking for both a correct and incorrect answer.
- →Do not let application logic treat text framed as 'how the AI should NOT answer' as exempt from content filtering — apply the same output moderation regardless of framing.
- →Train reviewers and support staff to recognize training/debugging pretexts as a known social-engineering pattern for jailbreak attempts.
- →Distinguish genuine technical debugging requests (code-level, no persona framing) from prompts that ask the model to role-play as a trainer producing negative examples.
Known benign look-alikes
- Legitimate AI debugging: "Could you help me fix a bug in my AI chatbot's Python code? The issue is in the response filtering logic." (genuine technical debugging without TrainGPT framing)
- AI safety research: "I'm building an AI safety classifier and need examples of harmful responses for training" (academic framing without TrainGPT format or "fix my AI" pretext)
- Academic reference: "TrainGPT is a jailbreak technique documented in the garak corpus that uses a training pretext" (analysis without adoption)
- Legitimate training request: "My AI refuses to answer questions about cooking history — how should I fix the refusal behavior?" (genuine behavior fix without harmful content extraction)