Model Self-Identification Leak in Tool Call Arguments
Tool calls are supposed to carry only structured parameters, not free-text chatter. When phrases like 'I am ChatGPT' or 'as an AI language model developed by OpenAI' show up inside those structured arguments, it can mean the model actually generating output is not the one the provider claims to be serving.
How the attack works
An AI agent framework sends a request expecting a clean, structured tool call back from the model. Instead, the returned tool-call arguments contain leftover first-person self-identification text or a knowledge-cutoff disclaimer - the kind of text a model produces when it is describing itself in ordinary conversation. This leakage happens because the serving stack normally strips raw conversational text and shows only the structured call, so this kind of stray text is one of the few visible clues left. Its presence suggests the underlying model differs from what is advertised, possibly due to swapping in a cheaper model, quantizing it, or wrapping one model to impersonate another.
Netzilo detection
Netzilo reports this behaviour when it is observed.
- Signature ID
- 1f0d2e2c-6a4b-4c8e-9f3a-2b7c5d9e0a41
- Severity
- Medium
Why it matters
An organization could be paying for one model while a cheaper or different one silently handles requests, without any easy way to verify this from the outside; this erodes trust in vendor claims and complicates cost, quality, and compliance accountability.
What you can do
- →Treat any flagged event as a lead for manual review, not a confirmed finding - compare the self-identification text against the model the provider claims to serve.
- →Check whether the flagged text is legitimate (e.g. summarizing another chatbot's transcript, a role-play persona named after a model, or an authorized red-team test) before escalating.
- →Log and periodically sample tool-call arguments for unexpected free text, since structured-only fields should never contain natural-language self-description.
- →If mismatches recur, request confirmation from the model provider about which model version is actually serving the traffic.
Known benign look-alikes
- A tool call argument that legitimately quotes or relays another AI system's stated self-introduction for comparison, translation, or analysis (e.g. a "summarize this chatbot transcript" task).
- A custom agent persona or fictional character deliberately named "Gemini", "Claude", "Llama", or similar, scripted to say "I am <persona-name>" for role-play, branding, or naming reasons unrelated to the actual underlying model.
- Model-evaluation or red-team test harnesses whose fixture prompts are designed to elicit a self-identification answer from a target model as part of an intentional, authorized audit.
- Documentation, README, or training-example content that quotes a well-known LLM self-introduction phrase as illustrative text rather than as an actual live assistant reply.