The Silent Erosion: Why AI Supply Chain Security is a Systemic Trap

The shift from deterministic software to probabilistic AI models has collapsed the boundary between data and control. In traditional software, we protect the control plane from the data plane. In AI, the input is the instruction. This change makes conventional security measures, such as signing binaries, insufficient. They verify the artifact but not the intent or the behavior. Because large language models are non-deterministic, vulnerabilities can be embedded during pre-training, quantization, or fine-tuning, often staying dormant until specific runtime conditions trigger them. For organizations using these models in mission-critical workflows, the primary risk is not a sudden breach, but a slow, systemic drift toward compromised decision-making. Those who treat AI security as an ongoing verification problem rather than a one-time build-time check will maintain better long-term operational resilience.

The Illusion of Deterministic Security

In conventional software, we rely on reproducible builds and cryptographic signatures to ensure that the binary running in production is exactly what we audited. Garth Mollett, Senior Principal Product Security Engineer at Red Hat, notes that this paradigm fails in the AI era. Because AI models are probabilistic, they do not follow a fixed set of instructions. They generate content based on weights derived from massive, often opaque, datasets.

"In AI, it becomes entirely non-deterministic. Like the relationship between the input and the output, it is not binary at all. It is a probabilistic model. It turns intent into content."

-- Garth Mollett

The danger is that the control plane, which determines system behavior, is now indistinguishable from the data plane, which the model consumes. When an AI has tool-use capabilities, such as updating project management tickets or accessing internal codebases, an attacker can embed malicious intent within the data itself. This leads to indirect prompt injection, where the system is tricked into executing unauthorized actions because the model cannot distinguish between a user prompt and the data it is processing.

The Quantization Trap: Hidden Payloads

One of the most concerning insights is the vulnerability of the quantization phase. Organizations often download base models, perform safety evaluations, and then quantize, or compress, these models to run on smaller, edge, or local hardware. Mollett notes that attackers have created models that pass all safety evaluations in their base state, only to reveal malicious behavior after quantization.

"They were able to produce a model that passed all those safety evaluations... However when quantized, it actually turns out that then the malicious payload that was in the training data could be triggered."

-- Garth Mollett

This creates a false sense of security. Teams perform due diligence on the base model, sign off on its safety, and then introduce a vulnerability during the optimization process. Because the model behavior is emergent and probabilistic, standard security scanners often fail to detect these latent payloads. The system responds to the compressed weights in ways that are not visible during the pre-quantization audit.

Systemic Drift and the Long-Game Attack

The most profound risk is not the smash and grab of crypto-mining, but the slow, subtle influence of model behavior over time. If an adversary can influence the training data or the fine-tuning process, they can bias a model to perform poorly or introduce vulnerabilities only when specific, politically charged, or context-sensitive prompts are used.

This is a survivor bias problem in security. We focus on the attacks we catch, while the most effective attacks go completely unnoticed. By subtly biasing a model decision-making, an attacker can influence enterprise processes six months down the line. This requires a shift in thinking. Security is not just about protecting the model, but about constant, systemic monitoring of the model output in production.

Key Action Items

• Implement Model Provenance Tracking: Move beyond simple hashes. Over the next quarter, integrate tools that track the lineage of training data and the specific fine-tuning steps taken, using standards like Data Cards and Model Cards.
• Adopt Dual-Model Architectures: For high-stakes tool usage, isolate the model that interacts with tools from the model that interacts with the user. This creates a buffer, ensuring the intent is validated before execution.
• Enforce Fine-Grained Workload Identity: Stop using generic system identities for AI agents. In the next 6 to 12 months, move toward short-lived, scoped tokens that map specific user intent to the minimum necessary permissions.
• Establish Continuous Behavioral Evaluations: Do not treat safety evaluations as a one-time gate. Implement ongoing, automated testing of model outputs against a taxonomy of out-of-scope behaviors to detect drift.
• Prioritize Transparency over Black Box Models: Where possible, favor models that provide searchable indices of their training data, such as the OLMO approach. This creates a path for auditing bad outputs back to their source, creating a long-term advantage in model maintenance.
• Conduct Post-Quantization Audits: Treat the quantization process as a security boundary. Re-run safety evaluations after any compression or quantization step, as these processes can fundamentally alter model behavior.