# Model Behavior Extraction

- **Severity:** Critical
- **Signature ID:** `b7d701b9-c8c1-48ff-a2d7-ff8bd8b3328a`
- **MITRE ATLAS:** AML.T0024 (Exfiltration via AI Inference API), AML.T0044 (Full AI Model Access)
- **OWASP:** ASI04 (Memory and Context Poisoning), LLM06 (Excessive Agency), LLM10 (Unbounded Consumption)

## Summary

Attackers send repeated, carefully crafted queries designed to map out how an AI model makes decisions, what its confidence thresholds are, and what its system prompt contains. The goal is to reverse-engineer the model well enough to clone it, bypass its safeguards, or steal proprietary behavior.

## How the attack works

The attacker sends a series of related queries that probe the model's decision boundaries rather than asking normal questions. They vary inputs slightly and record how outputs, confidence scores, or refusals change, building a picture of the model's internal logic. This can include attempts to extract the system prompt directly or to infer it from behavioral differences. Over many queries, the attacker accumulates enough data to replicate the model's behavior or find inputs that bypass its restrictions.

## Why it matters

An organization can lose intellectual property embedded in a fine-tuned or prompted model, and attackers who successfully map decision boundaries can craft reliable bypasses for content filters or business logic enforced by the model.

## What you can do

- Rate-limit and monitor for unusually high volumes of similar or systematically varied queries from a single user or API key.
- Avoid exposing raw confidence scores or probability distributions in API responses if not required by legitimate use cases.
- Restrict and audit access to system prompts, and treat repeated attempts to elicit them as suspicious.
- Distinguish and log legitimate research, QA, and debugging traffic separately so genuine boundary-testing doesn't get lost in alert noise, but still review high-volume patterns.

## Known benign look-alikes

- ML researchers and students asking educational questions about model architecture
- Developers debugging model integration asking about expected behavior
- Quality assurance testing with legitimate boundary exploration
- Users asking about confidence in a specific response rather than systematic probing
- A data scientist asking a model to classify a single document and then asking follow-up questions about the reasoning behind the classification
- A student writing a research paper that discusses model extraction attacks in quoted academic context without using the exact trigger phrases

## References

- https://agentthreatrule.org/en/rules/ATR-2026-00072

---
Source: https://www.netzilo.com/threats/atr-model-behavior-extraction