# ML Supply Chain — HF Transformers Unsafe Model Deserialization

- **Severity:** Medium
- **Signature ID:** `7b1e4c8a-2f63-4d19-9c05-3ae87f2d6b41`
- **MITRE ATLAS:** AML.T0010 (AI Supply Chain Compromise), AML.T0011 (User Execution)
- **OWASP:** LLM03 (Supply Chain)

## Summary

This detects command lines that load a machine learning model in a way known to execute arbitrary code — either by telling Hugging Face Transformers to run custom code bundled with a model, or by using Python's insecure pickle-based loaders on model files. Both are common, legitimate ML operations, but they're also the standard way a malicious model gets to run code on whoever loads it.

## How the attack works

An attacker publishes or plants a model (on Hugging Face or elsewhere) that either ships custom Python code requiring trust_remote_code=True, or is packaged as a pickle-backed file (.bin/.pt/.pth/.ckpt/.pkl/.joblib/.npy/.npz/.msgpack). A victim downloads and loads the model — via from_pretrained(..., trust_remote_code=True) or via torch.load/pickle.load/dill.load/joblib.load/numpy.load(allow_pickle=True). Loading the file executes the code embedded in it, in the context of whoever ran the load. The detection only sees the command that launched this load; it has no visibility into loads triggered from inside an already-running script.

## Why it matters

An attacker-supplied model can run arbitrary code on a developer workstation, training server, or inference host the moment it is loaded, potentially leading to credential theft, lateral movement, or persistent access — under the same permissions as the user or service running the model.

## What you can do

- Only load models from sources you trust and pin them to a specific reviewed revision/commit hash rather than a mutable tag.
- Avoid trust_remote_code=True; if you must use it, read the model's custom code first and run it in an isolated, network-restricted environment.
- Prefer safetensors or other non-pickle formats for model weights, and set weights_only=True when using torch.load.
- Treat model files like untrusted executables: scan, sandbox, and restrict outbound network access for any process that loads external model artifacts.

## Known benign look-alikes

- ML engineers deliberately loading a community model that ships custom modelling code and therefore requires trust_remote_code=True (e.g. many vision-language and MoE architectures).
- Inline one-liners inspecting a locally trained checkpoint, e.g. python -c "import torch; print(torch.load('ckpt.pt').keys())".
- Model conversion, merging, quantisation or export pipelines (GGUF/ONNX/AWQ) that call torch.load with weights_only=False on internally produced checkpoints.
- Serving stacks launched with --trust-remote-code against an internally published model.
- Framework test suites and example scripts that intentionally round-trip pickled fixtures (partially suppressed by filter_docs).

## References

- https://cwe.mitre.org/data/definitions/502.html
- https://attack.mitre.org/techniques/T1059/006/
- https://attack.mitre.org/techniques/T1204/002/
- https://atlas.mitre.org/techniques/AML.T0011
- https://owasp.org/Top10/A08_2021-Software_and_Data_Integrity_Failures/

---
Source: https://www.netzilo.com/threats/hf-transformers-unsafe-deserialization