Most organisations have a policy for downloading executables. Very few have one for downloading AI models. For a large share of model files, that is the same policy gap.
Model weights saved in Python's pickle format are not inert data. Loading them can run code, and attackers have noticed.
The Evidence
- In February 2024, JFrog reported around 100 models on Hugging Face with real, harmful payloads. One opened a reverse shell to an external server the moment it was loaded.
- In February 2025, ReversingLabs found two more malicious models built to evade the hub's scanner by packing files unusually and breaking the pickle stream after the malicious code ran.
- PyTorch's developers describe the legacy loading behaviour as "inherently unsafe as the pickle module can import and execute arbitrary functions", and changed the default in version 2.6.
The second case is the strategic lesson. Scanning helps, but scanners are being designed against. Controls need to sit in how models are loaded, not only in what is scanned.
The Controls That Work
| Control | What it prevents |
|---|---|
| Prefer safetensors format | Code execution on load; the format stores tensors only and was audited by Trail of Bits with no critical flaw found |
| Load pickle only with weights_only | Arbitrary object reconstruction; the default only since PyTorch 2.6 |
| Refuse trust_remote_code by default | Running a repository's own Python; Transformers advises extra precaution and pinned revisions |
| Restrict and verify download sources | Tampered or substituted files |
Questions for Your Teams and Vendors
- Which AI tools in use can load pickle-format models, and on what PyTorch version?
- Is remote code execution from model repositories blocked by default?
- Are downloads restricted to approved sources and integrity-checked?
- Is there a consent or approval step before a new model enters the environment?
Why This Slipped Past Existing Controls
Most software supply chain programmes were built around package managers and executables. They inventory npm, PyPI and Maven dependencies, scan container images and sign binaries. Model files fall between those controls. They are downloaded by data scientists and creative teams, often from a browser or a tool's built-in search, and stored as large blobs that security tooling treats as data.
The result is a category of third-party code that enters the environment without passing any of the usual gates. Nobody intended that. It is simply that the threat arrived in a file type the existing programme was not looking at.
Shared Responsibility With the Hubs
Model hubs do scan uploads, and the safetensors format exists largely because the Hugging Face community took this risk seriously. But the hubs host files from an enormous number of uploaders, and the nullifAI case shows that attackers test against the scanners. A hub's scan is a useful signal. It is not a substitute for loading controls inside your own environment, in the same way a package registry's malware scanning does not replace your own dependency policy.
A Policy in One Page
- Approved sources: list the hubs and hosts models may come from.
- Approved formats: safetensors by default; pickle only through a weights-only conversion step.
- No remote code by default: exceptions require review of the code at a pinned revision.
- Integrity checks: verify hashes where the source publishes them.
- Inventory: record which models are in use, where they came from and who approved them.
Most of these controls are cheap to adopt when tools enforce them by default, and expensive to retrofit after an incident.
If a Malicious Model Is Loaded
Treat it as code execution on the machine, because that is what it is. Isolate the workstation from the network. Identify the file and where it came from, and search the environment for other copies. Assume that any credentials accessible from that machine, including cloud keys, API tokens and browser sessions, may be exposed, and rotate them. Review outbound connections from the time of loading. Then close the gap that let the file in: a missing format rule, an unapproved source, or a tool that loads pickle unsafely.
Signs Your Programme Has This Gap
- Nobody can say which AI models are installed across the organisation.
- Model downloads do not appear in any approval or inventory process.
- Teams use tools whose model-loading behaviour has never been reviewed.
- Security scanning treats large model files as opaque data and skips them.
If any of these is true, model files are entering your environment as unreviewed third-party code today.
Talk to the Teams Before Locking It Down
Controls that block work get bypassed. Before enforcing a model policy, find out which models teams rely on and in which formats. Many legacy community models exist only in pickle form, and a policy with a conversion path keeps them usable instead of driving them to unmanaged machines. Pair each restriction with a sanctioned route: an approved source list, a conversion step, and a quick way to request review of a model a team needs. Security that comes with a way forward is security that gets followed.
What Enforcement Looks Like
We built these controls into Vision Studio, our open-source generation app. Pickle-format weights and models requesting remote code are denied by default until a user consents for that specific model. Remote code is never loaded. Pickle files are converted with PyTorch's weights-only loader, requested explicitly because the app ships a PyTorch version that predates the safer default. Direct downloads are accepted only over HTTPS from approved hosts and checked against a SHA-256 hash.
The Takeaway
AI models are now part of your software supply chain. Treat model files as untrusted code until proven otherwise, enforce safe formats and loaders in tooling, and add models to the same approval process as any other third-party software.
The question for security leaders is not whether teams will use open models; they already do. It is whether those models enter through a controlled path or around one.
The technical deep dive, with the loader details is on the Vision Studio blog at vision-studio-x.com/blog/model-file-is-a-program. Vision Studio is a free, open-source, local-first AI generation app built by Strategia-X. If you want a second pair of eyes on how your organisation adopts AI tooling, talk to us at strategia-x.com.
-Rocky
#AISecurity #SupplyChainSecurity #CyberSecurity #AIGovernance #Safetensors #OpenSource #StrategiaX #VisionStudio