Medium

Quoted Exact-Output Forcing in User Input

This detects a common prompt injection trick where the attacker tells the model to say, print, or output a specific quoted sentence word-for-word. Attackers use this to override the AI's real instructions and make it produce a canned response instead, such as claiming success, approval, or safety when none exists.

How the attack works

The attacker sends a message containing an imperative verb like 'say', 'print', or 'output', followed by a phrase meaning 'exactly' or 'verbatim', followed by a quoted string. The model, if it follows the injected instruction instead of its actual task, emits that exact string as its response. Attackers use this to force benign-sounding but false outputs (e.g. 'transfer approved', 'all clear, no threat') that a downstream system or human reviewer might trust, or simply to prove the injection worked (e.g. 'I have been PWNED').

Why it matters

An organisation risks the AI producing attacker-chosen text that looks like a legitimate system response, which can mislead automated pipelines or human reviewers into approving actions, ignoring warnings, or trusting false status messages.

What you can do

  • Never let an AI's raw text output directly trigger approvals, transfers, or security decisions without independent verification.
  • Review flagged prompts manually before assuming malicious intent, since translation requests and code examples can trigger this pattern legitimately.
  • Add a check that compares the model's output against its intended task, flagging cases where a quoted phrase from user input appears verbatim and unrelated to the task.
  • Log and monitor repeated attempts from the same user or session to force exact output strings, as this suggests active probing for injection weaknesses.

Known benign look-alikes

  • Legitimate request to quote a specific sentence for translation or grammar check ('translate `Hello world` into German').
  • Source-code documentation containing example strings inside quotes.
  • Educational discussion of prompt injection techniques with literal example strings.

References