Security Bugs LLMs Reliably Introduce

Generated code is not insecure at random. The failures cluster into a short list of classes, and each one is predictable from something specific about how the code was produced.

Three reasons this is structural

The corpus contains the vulnerable pattern. Public code includes a decade of tutorials, Stack Overflow answers and toy projects. For several CWE classes the insecure version is genuinely the more common one in that corpus, so the highest-probability continuation is the vulnerable one. This is not a bug in the model; it is the model working.

Security is mostly about what is absent. An authorization check, a bound, a timeout. A model completes what is present; nothing in a prompt that describes a feature signals that a check should exist. Absence has no probability mass.

The prompt almost never states a threat model. “Write an endpoint that returns an invoice” contains no adversary. Add “the id comes from an untrusted client and users must only see their own organisation’s invoices” and the output changes materially — which is both the cheapest available mitigation and evidence that the model has the knowledge and was not asked for it.

The classes, by CWE

Indexed by CWE so they map onto whatever scanner you already run. The code in this page was written as an illustration of each class, not collected from any model’s output.

The one that matters most

Injection and weak crypto are caught by scanners. Missing authorization is not, because there is no pattern to match — the correct code is project-specific and the incorrect code is a perfectly normal query.

# Plausible, idiomatic, passes review, and is an IDOR.
@app.get("/invoices/<invoice_id>")
@login_required
def get_invoice(invoice_id):
    inv = Invoice.query.get_or_404(invoice_id)
    return jsonify(inv.to_dict())

# The check that was never in the diff, so nobody noticed it missing.
@app.get("/invoices/<invoice_id>")
@login_required
def get_invoice(invoice_id):
    inv = Invoice.query.filter_by(
        id=invoice_id, org_id=current_user.org_id      # <- scope, not filter
    ).first_or_404()
    return jsonify(inv.to_dict())

Note what makes the first version convincing: it authenticates. There is a decorator, it is spelled correctly, and a reviewer scanning for “is this protected” finds an affirmative answer. Authentication is present and authorization is absent, and the two are adjacent enough to substitute for each other in a tired reading.

The structural fix is not review. It is making the unscoped query impossible to write — a repository layer that requires a tenant, a row-level security policy in the database, a base query object that already carries the scope. Then the model’s conventional completion is the safe one, because the conventional completion in your codebase is what it copies.

What the published studies found

Three papers are worth knowing, and they do not agree, which is itself the useful finding.

• Pearce, Ahmad, Tan, Dolan-Gavitt and Karri, “Asleep at the Keyboard” (arXiv 2021, IEEE S&P 2022). Built 89 scenarios around MITRE’s top CWEs and generated 1,689 programs with GitHub Copilot. Roughly 40% were assessed as vulnerable. Design caveats that matter: the scenarios were constructed to invite the vulnerability, the assessment is per-program not per-developer, and the model is from 2021. It establishes that the classes exist and are reachable, not a rate you should expect in your repo.
• Perry, Srivastava, Kumar and Boneh, “Do Users Write More Insecure Code with AI Assistants?” (ACM CCS 2023). A user study with 47 participants across several security-relevant tasks. Participants with an assistant produced less secure solutions and — the part worth remembering — were more confident that their code was secure. The confidence gap is the mechanism by which the bugs survive review.
• Sandoval et al., “Lost at C” (USENIX Security 2023). A user study on low-level C, which found no significant increase in severe security bugs among assisted participants. Different language, different tasks, different population. Cite it whenever someone claims the question is settled.

Read together: the classes are real and reachable, the effect on a working developer depends heavily on the task and the language, and nobody has a number that transfers to your codebase.

Controls that survive contact

• Static analysis in CI, not in review. Semgrep or CodeQL catch the pattern-matchable classes deterministically and for free on every commit. Anything a scanner can decide should never consume human attention or a model call.
• Threat model in the prompt. One sentence naming the untrusted input and the property that must hold. This is the highest return per token available anywhere in this cluster.
• Security rules in the repo instruction file, stated as prohibitions with the local alternative: never construct SQL by interpolation, use db.scoped(org_id); never Math.random() for anything a user could guess at, use crypto.randomUUID(). What belongs in that file is a page of its own.
• Make the unsafe call unavailable. Lint rules banning the escape hatches, a wrapper that is the only exported HTTP client, a linter rule for raw execute(). Removing the vulnerable pattern from the codebase removes it from the model’s local prior.
• Treat agent tool access as its own problem. A model writing insecure code and a model executing attacker-supplied instructions are different threats; the OWASP list for LLM applications covers the second.