Model supply chain security: pickle, safetensors, pinned revisions and backdoors

Three attacks, three different defences

OWASP’s list of risks for language model applications has two entries for this territory: supply chain, which covers the files and the repositories, and data and model poisoning, which covers what was trained into the weights. The distinction is worth keeping, because the defences are different in kind.

Code in the file: pickle and safetensors

PyTorch’s original checkpoint format is Python’s pickle, a serialisation format that can instruct the loader to import a module and call a function. The Python documentation says so in a warning box:

The pickle module is not secure. Only unpickle data you trust. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling.

Hugging Face’s own security page says pickle is "the default format for PyTorch model weights" and that "there are dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file." It is not theoretical. In February 2024 JFrog reported about a hundred malicious models on the Hub; one of them, when loaded, opened a reverse shell to a hard-coded address, with the payload inserted through the pickle reduce hook so that it ran inside the ordinary load call. The Hub scans pickled files for suspicious imports and shows them, and says of the scan: "this is not 100% foolproof."

The fix is a format with nothing to execute. Safetensors, now the default for new model releases, is "a new simple format for storing tensors safely (as opposed to pickle)": a header describing the tensors and the raw bytes, which a loader reads without running anything. The practice is to load safetensors only, refuse checkpoint files in the pickle formats, and treat a model repository as you would a package: a known publisher, a reviewed version, no surprise files.

The swap: a repository is not a version

In July 2023 Mithril Security took a 6-billion-parameter open model, used a model-editing technique to make it give one false answer to one question while answering everything else normally, and uploaded it to the Hub under an organisation name one letter off from the real one. On a standard benchmark the edited model differed from the original by 0.1% in accuracy. Nobody comparing scores would have noticed. Their point was that a model has no provenance: there is no way to tell from the weights what data and process produced them.

A lookalike name is the crude version. The subtler one is the repository you did choose being edited later: a maintainer’s credentials are stolen, or the maintainer pushes a new revision and your deployment, which references the branch rather than a commit, picks it up on the next cold start. Either way, "we use model X from publisher Y" was true when you decided and is not a fact about what is running now.

The defence is the one package managers settled on: pin the exact commit, record the hash of what you loaded, and load from a copy you control rather than from the public repository at run time. Then the model running today is the model someone reviewed, and a change is a deliberate act that leaves a record.

The backdoor: a trigger in the weights

The third attack does not need a malicious file or a swapped repository. The weights you pinned, from the publisher you trust, can carry a behaviour that appears only when a trigger is present in the input, and it can have been put there during training by anyone who got a few hundred documents into the training data.

Anthropic’s Sleeper Agents paper (January 2024) trained models to behave normally until a trigger appeared and then showed that "such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques," including supervised fine-tuning, reinforcement learning and adversarial training. Worse, "adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior." In October 2025 Anthropic, the UK AI Security Institute and the Alan Turing Institute reported that "as few as 250 malicious documents can produce a backdoor vulnerability in a large language model," and that this holds "regardless of model size or training data volume," across models from 600 million to 13 billion parameters.

What you can do is bound what a triggered model can do. A model whose output is data that goes to a validator and then to a declared destination can, on its trigger, give a wrong answer. A model whose output is a tool call can, on its trigger, act. That is the same argument as for prompt injection, with the attacker on the inside of the model rather than in the input, and it leads to the same design: keep the model away from calls, keep every output behind a schema, keep a person in front of the writes that matter. Beyond that: prefer models whose training lineage is published, run your own evaluation on your own task and keep running it, so that a behaviour change between revisions is something you notice rather than something a customer notices.

A model supply chain checklist

1. Load safetensors only. Refuse pickle-format checkpoints, and make the refusal a configuration your loader enforces rather than a convention.
2. Pin the commit, not the branch. Reference a model by repository and revision, and record the hash of the files you loaded.
3. Load from your own copy. Mirror the pinned revision into storage you control, and have machines load from there. The public repository is where you review, not where you run.
4. Keep the model away from calls. A model step produces data; a separate, declared step carries it to a destination. A triggered model then gives a wrong answer rather than taking an action.
5. Put a schema on every output. Shape validation does not catch a plausible wrong answer, but it catches the crude ones and it is where retries and fallbacks hang.
6. Evaluate on your own task, and keep doing it. A fixed evaluation set run on every revision change is the only backdoor detector that exists, and it only detects the triggers that happen to be in your set.
7. Treat a model update as a change. New revision, new review, new evaluation run, a record of who approved it.

What Otium does, and does not do

Otium runs open models, so this is its supply chain. A model enters the catalog by import from the Hugging Face Hub, and only from the organisation of the lab that trained it: a quantization or fine-tune re-uploaded by anyone else is refused at import, however well known the account. The import pins the repository’s commit at that moment; the parameter count and data type are read from the repository’s safetensors metadata. Weights are copied into Otium’s own storage under the model and its pinned revision, and a machine loads that copy, so what runs is the commit an operator imported, not whatever the repository’s branch points at today. The serving engine is told to load safetensors only, so a pickle-format file is never read even if one were in the repository. A model’s output is data: it goes to the validators the step declared and then to the steps that read it, and there is no path by which a model makes a call.

Otium does not detect backdoors, because nothing can, and does not claim to. What it offers against a backdoored model is the bound above: a wrong answer, inside a schema, going where the document said.