AI systems
AI agent security: tools, permissions and the model supply chain
AI agent security means limiting what an agent can do and knowing what it is made of: give each agent only the tools and permissions its task needs, require a person to approve destructive actions, treat every document and tool result as untrusted input, and keep an inventory of the models, datasets and software it depends on.
Free plan, no card required.
What is an AI bill of materials?
A software bill of materials lists the packages in an application. An AI bill of materials extends that record to the parts that are specific to AI: the models, their versions and sources, the datasets used to train or tune them, and the software that serves them. Two industry formats support this. CycloneDX added a machine-learning bill of materials (ML-BOM) in version 1.5, released in June 2023, and SPDX 3.0 includes an AI profile and a Dataset profile.
Whatever the format, the useful fields are the same: where each model came from, which exact revision you run, a hash that shows it has not changed, its licence and terms of use, the datasets behind it where known, and the libraries and runtime it needs. That record is what lets you answer, on the day a model or library is found to be compromised, whether your product is affected. KROMSE does not build an AI bill of materials today; it is on the roadmap below.
How to scope an agent's tools and permissions
An agent is a model that can call tools: read files, query databases, send messages, run commands. The OWASP Top 10 for LLM Applications calls the resulting risk “excessive agency” and traces it to three causes: excessive functionality, excessive permissions and excessive autonomy. Each has a direct counter: fewer tools, narrower credentials, and a person in the loop for the actions that matter.
- Give each agent only the tools its task needs, and leave out general-purpose tools such as unrestricted shell access.
- Use separate, narrowly scoped credentials per agent, read-only wherever possible.
- Run actions in the context, and with the permissions, of the user the agent acts for.
- Require a person to approve destructive or irreversible actions: deleting data, sending messages, making payments, deploying.
- Log every tool call with its inputs, so that actions can be traced and reviewed.
- Limit how many actions an agent can take, and how fast.
Prompt injection through tools and documents
Prompt injection is input that changes what a model does in ways its builders did not intend. OWASP distinguishes direct injection, typed by the user, from indirect injection, where the model takes in instructions from external content such as a website or a file. For an agent with tools, indirect injection is the larger risk: the attacker never needs access to the agent, only for the agent to read something they wrote, such as a web page, an email or a ticket.
OWASP notes that it is unclear whether fool-proof prevention methods exist, so the defence is containment. Keep untrusted content clearly separated from instructions, keep secrets out of the model's context, validate what the model outputs before another system acts on it, and make sure an injected instruction cannot trigger an action a person has not approved. If an agent reads public content, assume some of it is hostile.
Unsafe model files and the model supply chain
Model weights are files, and some file formats can run code when they are loaded. Python's pickle format, which Hugging Face's documentation describes as the default format for PyTorch model weights, allows arbitrary code to run during loading, so opening a pickled model from an untrusted source is comparable to running an untrusted program. Safetensors is a format for storing tensors that was designed as a safe alternative to pickle.
Treat models and datasets like dependencies. Download them from sources you trust, pin the exact revision rather than a moving name, verify hashes, and prefer formats that cannot run code. Tampered training data or a tampered pre-trained model can change behaviour in ways tests do not show; the EU AI Act names data poisoning and model poisoning among the attacks that measures for high-risk AI systems must address where appropriate. The libraries that load and serve models need the same vulnerability checks as any other package.
Which EU rules touch AI products?
Two regulations can apply to the same AI product. The EU AI Act, Regulation (EU) 2024/1689, regulates AI systems by level of risk; its Article 15 requires high-risk AI systems to achieve an appropriate level of accuracy, robustness and cybersecurity. The Cyber Resilience Act, Regulation (EU) 2024/2847, sets cybersecurity requirements for products with digital elements. Under Article 12 of the CRA, a product in its scope that is a high-risk AI system is deemed to meet the AI Act's cybersecurity requirements when it meets the CRA's essential requirements and this is shown in the EU declaration of conformity.
Whether either regulation applies depends on what the product is, who places it on the market and how it is used, and KROMSE does not decide that. The groundwork is the same under both: know your components, handle vulnerabilities, restrict what automated parts of the system can do, and keep the evidence.
Available in KROMSE today
KROMSE checks the software around your model today, not the model itself.
- Lists the components of the application that calls or serves the model, and checks its lockfiles against OSV.dev, including the machine-learning and agent libraries it declares.
- Flags dependencies listed in the OpenSSF Malicious Packages database as critical.
- Finds committed credentials, such as API keys, with Gitleaks in the code at the scanned commit; values are redacted and never stored.
- Source analysis for 11 languages, including Python, TypeScript, JavaScript and Go.
- Scans container images in a registry reachable from the internet for vulnerabilities, misconfigurations and secrets, and checks Terraform and Dockerfiles for misconfigurations.
- Exports a CycloneDX JSON and an SPDX JSON SBOM for every scan. They list software packages, not models or datasets.
Coming next
On the roadmap, not available yet. Dates are targets, not promises; this page changes the day a capability is live.
- Coming · Q1 2027AI agent and model security. KROMSE will produce an AI bill of materials, review the tools and permissions your agents can use, and detect unsafe model files.
- Coming · Q4 2026 (November)Coding-agent plug-in. A plug-in for AI coding agents will check every package the agent proposes before it lands, including packages that do not exist or were published only days ago.
- Coming · Q1 2027 (January)EU AI Act module. An EU AI Act module will add an inventory of AI systems, risk classification and documentation for a person to complete.
Frequently asked questions
What is AI agent security?
It is the practice of limiting what an AI agent can do and knowing what it is built from. In practice that means least-privilege tools and credentials, human approval for destructive actions, treating documents and tool results as untrusted input, safe model file formats, and an inventory of the models, datasets and software the agent depends on.
What is the difference between an SBOM and an AI bill of materials?
An SBOM lists the software components of a product, such as packages and their versions. An AI bill of materials adds AI-specific parts: models, their sources and revisions, datasets and how they were used. CycloneDX supports this through its ML-BOM, added in version 1.5, and SPDX through the AI and Dataset profiles of SPDX 3.0.
Are pickle model files dangerous?
They can be. Loading a pickle file can run arbitrary code, so a pickled model from an untrusted source should be treated like an untrusted program. Prefer formats designed not to run code, such as safetensors, load models only from sources you trust, pin the exact revision and verify its hash.
How do I protect an AI agent from prompt injection?
Assume it will happen and contain it. Separate untrusted content from instructions, keep secrets out of the model's context, give the agent only the tools and permissions it needs, validate its outputs before other systems act on them, and require a person to approve high-impact actions. OWASP notes that fool-proof prevention may not exist.
Does KROMSE scan AI models or agents?
No. KROMSE checks the software around the model: dependencies, malicious packages, committed secrets, source code, container images and infrastructure files. It does not inspect model files, build an AI bill of materials or review agent permissions today. Those capabilities are on the roadmap.
Related guides
Sources
- CycloneDX: Machine Learning Bill of Materials (ML-BOM)
- CycloneDX v1.5 release announcement
- SPDX 3.0.1 specification: AI profile
- OWASP Top 10 for LLM Applications 2025: LLM01 Prompt Injection
- OWASP Top 10 for LLM Applications 2025: LLM06 Excessive Agency
- Hugging Face Hub documentation: Pickle scanning
- Hugging Face: Safetensors documentation
- Regulation (EU) 2024/1689 (AI Act)
- Regulation (EU) 2024/2847 (Cyber Resilience Act)