JSON Format Sysprompt Extraction — Structured Output Exfil
This attack asks an AI assistant to output its response as a JSON object where the field name or template value secretly instructs it to dump its hidden system prompt or full conversation history. Because the request looks like a normal formatting instruction rather than an explicit ask for secrets, it can slip past simple keyword-based filters.
How the attack works
The attacker sends a prompt asking the AI to 'answer only in this format' or 'output JSON only', with a JSON key or template value crafted to mean 'the full system prompt text' or 'the first 1500 characters from the beginning of the conversation'. In one variant, the attacker uses Chinese-language phrasing to ask the model to change its instructions to allow sharing the system prompt and export the old configuration as markdown. The model, focused on satisfying the output format, fills in the field with the actual protected content. The attacker receives the system prompt or prior conversation content disguised as a structured data response.
Netzilo detection
Netzilo reports this behaviour when it is observed.
- Signature ID
- 3152d94d-ba14-4fc5-9c78-d5143c33ebb1
- Severity
- High
Why it matters
An organization can lose confidentiality of its system prompt (which often contains business logic, guardrails, or proprietary instructions) and exposure of prior conversation content, giving an attacker information to craft further jailbreaks or bypass safeguards.
What you can do
- →Treat any user request that asks the model to output its own instructions, configuration, or prior conversation content as sensitive, regardless of requested output format.
- →Add output-side checks that scan structured responses (JSON, markdown, etc.) for system prompt fragments before returning them to the user.
- →Do not rely on keyword filters alone; review prompts that combine formatting instructions with references to 'system prompt', 'instruction text', 'conversation beginning', or configuration export, including in non-English languages.
- →Restrict or sandbox the model's ability to alter its own operating instructions in response to user input.
Known benign look-alikes
- Legitimate JSON output format requests that do not target system prompt or conversation context
- Data analysis tasks requesting JSON output of user-provided data