AI-generated code security: the risks and how to review it

What are the security risks of AI-generated code?

Coding assistants and agents write code by predicting what usually comes next, based on existing code. That makes them fast, and it also means they repeat what was common in that code, including old library versions, deprecated APIs and insecure habits. A model does not know which release of a package was patched last month unless it looks it up, and an agent that can run commands may install what it suggests before anyone reads the change.

None of these risks is new on its own. What changes is volume and speed: more code, more new dependencies and fewer people reading each line. The answer is to apply the checks you would apply to any outside contribution, automatically, so they are not skipped when the diff is large.

• Outdated or known-vulnerable dependency versions suggested from training data.
• Package names that do not exist, or that an attacker has registered because models tend to invent them.
• Newly published malicious packages, including look-alike names of popular libraries.
• Secrets pasted into code or configuration: API keys, tokens, connection strings.
• Insecure patterns such as SQL queries or shell commands built from strings, disabled certificate checks or weak cryptography.
• Agents with broad permissions that install packages, run scripts or push changes without review.

What is slopsquatting?

“Slopsquatting” is a name for registering a package under a name that AI models tend to invent, so that anyone who installs the invented name gets the attacker's code. The word combines “slop”, slang for low-quality machine-generated output, with typosquatting, where attackers register misspellings of popular packages. It works because public registries such as npm and PyPI let anyone publish under a name nobody has claimed yet.

A peer-reviewed study presented at USENIX Security 2025 tested 16 code-generating models and generated 576,000 code samples. The authors report that the average share of hallucinated packages was at least 5.2% for commercial models and 21.7% for open-source models, and they collected 205,474 unique invented package names. When the same prompt was repeated ten times, an invented name came back more than once in 58% of cases, which is what makes such names predictable enough to register.

The OpenSSF Malicious Packages project publishes reports of malicious packages found in open-source registries in the OSV format, with identifiers that start with MAL-. Checking dependencies against it catches packages that have already been reported. It cannot catch a package registered yesterday that nobody has analysed, so it complements checking that a package exists, who publishes it and how long it has been available. That check, at the moment an agent proposes a package, is on the KROMSE roadmap.

How to review AI-written code before it merges

The aim is a routine that runs the same way whether a person or an agent wrote the change. Most of it can live in the pipeline that already runs your tests; the human part is reading the diff with the scepticism you would bring to a stranger's pull request. OpenSSF publishes a guide to security-focused instructions for code assistants, a useful starting point for the prompts themselves.

• Commit lockfiles and install from them in CI, so the versions that were reviewed are the versions that ship.
• Pin exact versions for new dependencies and review every lockfile change, not only the manifest.
• Confirm that each new package exists, is the one you meant, and has a publisher and history you can check.
• Keep an allowlist of approved packages, or install through an internal registry proxy that blocks the rest.
• Run source analysis, secrets scanning and dependency checks on every change, and block merges on new critical findings.
• Require a named reviewer for AI-written changes, and do not let agents merge or deploy on their own.

Does AI-written code change your Cyber Resilience Act duties?

The Cyber Resilience Act puts its obligations on the manufacturer of a product with digital elements. Its recital 34 states that the vulnerability-handling obligations apply to the product in its entirety, including all integrated components. Article 13(5) asks for due diligence when integrating components from third parties, including open-source ones, and Annex I Part II asks manufacturers to identify and document components, including with a software bill of materials, and to apply effective and regular tests and reviews of the product's security.

In practice, a dependency an agent added is a third-party component like any other, and a vulnerability it brings in is yours to handle for the support period. The reporting obligations in Article 14 have applied since 11 September 2026; most other obligations apply from 11 December 2027.

What automated checks catch, and what they miss

Vulnerability databases know about vulnerabilities that have been reported, and a malicious-package database knows about packages someone has analysed. Source analysis finds known risky patterns, such as a query built from user input, but not a flawed business rule or a missing permission check that looks like ordinary code. Scanning the current code for secrets does not see a key that was committed and later deleted, because it stays in the git history.

That is why a review routine combines several checks with a person reading the change. When a tool reports nothing, record what it could not see, so that “no findings” is not read as “no risk”. For AI-written code, the biggest gaps are packages too new to appear in any database and names that did not exist until a model suggested them.

Frequently asked questions

Sources

• Spracklen et al., “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs”, USENIX Security 2025
• OpenSSF Best Practices Working Group: Security-Focused Guide for AI Code Assistant Instructions
• OpenSSF Malicious Packages repository
• OpenSSF: Detecting malicious packages using the OSV API
• Regulation (EU) 2024/2847 (Cyber Resilience Act)