Your CI dashboard is green. Your release notes are written. Your build passed every gate you designed. And the product your customer installs does not start.
That happened to us on 13 September. One commit went to three build machines. Two produced working applications. The third produced a bundle that died on import, from identical source, with an identical build script. Nothing was attacked. No package was malicious. Every dependency behaved exactly as documented.
The failure is worth writing up because the governance gap it exposes is not specific to us, and most engineering organisations have the same one open right now.
What Actually Happened
Our installer requested three Python packages with no version attached: torch torchvision torchaudio. On Windows and Linux that request resolves against the CUDA 12.1 wheel index, and that index stops at torch 2.5.1. An open request against a frozen index returns the same answer every time. Those platforms were not pinned. They were accidentally pinned by an upstream that had stopped moving.
macOS has no CUDA wheels. It falls through to the CPU index, and that index tracks the latest release. The same unpinned line that quietly meant "2.5.1" on two platforms meant "whatever shipped this week" on the third. In September that was torch 2.14.0, and the resulting bundle raised RuntimeError: operator torchvision::nms does not exist before rendering anything.
Three releases had shipped with that line in place. It had never failed, because on two of three platforms it could not.
The Governance Lesson, Not the Python Lesson
Strip out the specifics and the shape is familiar to anyone who has run a release process:
- A control existed on paper. The build had a defined dependency spec.
- The control was not actually enforcing anything. The spec named packages, not versions.
- An unrelated external condition was doing the enforcement instead. An upstream index that had stopped publishing.
- The control failed the moment that external condition changed, in the one place it did not apply.
This is the same failure mode as a disaster recovery plan that has never been tested, or an access review that rubber-stamps whatever the system exports. The artefact exists. The assurance does not.
The industry data suggests this is close to universal. Datadog's 2026 research found 71% of organisations never pin a CI action to an immutable commit SHA, relying on mutable tags that an upstream can rewrite. The instinct that leaves a CI tag floating is the same one that leaves a runtime dependency floating, and both are invisible until an upstream moves.
The Second Finding: Your Audit Is Reading the Wrong List
The same release carried a security fix that makes a sharper governance point.
GHSA-2883-xcg3-v3hh (CVE-2026-84375, CVSS 7.5, High) is a denial of service in js-yaml, a YAML parser. On the face of it, a YAML parser is build tooling. Most teams would triage it as low priority and move on.
In our case it was not build tooling. It reached the shipped desktop application through electron-updater, which parses the update feed the installed product fetches over the network. That is remote input, parsed by the binary on the customer's machine, on a schedule. The severity did not change. The exposure did, completely.
Standard dependency audits report your dependency tree. They do not report your shipped tree. Those are different lists, and only one of them is reachable by someone who does not work for you. If your vulnerability process cannot tell you which finding sits inside the artefact you distribute, it is prioritising by the wrong axis.
Why This Is Not the Supply Chain Story You Have Been Reading
Board-level conversation about software supply chain risk is dominated by malicious packages, and the numbers justify the attention. Sonatype's 2026 report counts more than 1,233,000 malicious open source packages catalogued across npm, PyPI, Maven, NuGet and Hugging Face, and incident frequency has been climbing steeply year over year.
Neither of our two findings was malicious. One was a resolver answering an open question correctly. One was an honest CPU bound that missed an edge case. Both shipped broken or exposed software to customers anyway.
A supply chain programme built only to detect hostile packages will not catch either. The uncomfortable version: most of the defects that will actually reach your customers this year will come from dependencies that are behaving perfectly.
Four Questions Worth Asking Your Engineering Leads
| Question | What a weak answer sounds like | What you want to hear |
|---|---|---|
| What does our build resolve, not request? | "It is in the dependency file." | A pinned manifest, and a record of what shipped. |
| Which dependencies are in the artefact we distribute? | "The audit is clean." | A distinct shipped-tree inventory, triaged separately. |
| Does a green build on one platform tell us about the others? | "CI passed." | Per-platform evidence, and a named gate on each. |
| What enforces our build rules? | "It is documented." | A test that fails the build. Documentation is not a control. |
What We Changed
All four install paths now pin one exact set: torch 2.5.1, torchvision 0.20.1, torchaudio 2.5.1. Those versions were not chosen for being newest. They are the exact set that the Linux machine built, smoke tested and published on the same run the macOS machine failed. The version that shipped is the version we pinned.
More importantly, a test now fails the build if any install path is left free to resolve its own version again, and a second test asserts the shipped update path stays out of the advisory range and still parses a real production feed correctly. A rule written in a comment is a suggestion. A rule that fails the build is a control.
It is also worth noting what PyTorch had already told us: they stopped building macOS x86_64 binaries after version 2.2.0 in January 2024, and published the deprecation RFC ahead of it. The platform asymmetry that broke us was public information. We had simply never had a reason to look, because nothing had failed yet.
The Decision You Deferred
Every unpinned dependency is a decision you have handed to a resolver, to be made on a date you will not choose, on a machine you may not be watching. Most days it makes the same choice it made yesterday, and you never find out the decision was open at all.
You find out on the day a customer opens your product and it does not start.
The full engineering post-mortem, including the resolver behaviour and the exact fixes, is on the Vision Studio blog at vision-studio-x.com/blog/same-commit-two-builds. Vision Studio is a free, open-source, local-first AI generation app built by Strategia-X. If you want a second pair of eyes on your own release and dependency governance, talk to us at strategia-x.com.
-Rocky
#DependencyManagement #SupplyChainRisk #ReleaseEngineering #TechnicalGovernance #BuildReliability #OpenSource #EngineeringDreams #StrategiaX #VisionStudio