Threat Modelling an AI Feature
4 min read · updated August 3, 2026
You do not need a new methodology for AI features. You need the one you have, applied to a system where a component reads attacker-authored text and emits instructions to other components.
Why STRIDE needs adapting at all
STRIDE — spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege — assumes you can draw a data flow diagram and mark where trust changes. That still works. Two of its assumptions do not survive contact with a model, and naming them is the whole adaptation:
- A process transforms data; it does not obey it. In a classic diagram, a component processes input according to its code. A model processes input according to input. Every data flow into a model context is therefore also a control flow.
- Output has a known shape. Classic components emit structured results you can constrain. A model emits arbitrary text whose content may have been chosen by whoever wrote the data it read, so its output edge inherits the trust level of its worst input.
Everything else — the categories, the diagram, the mitigation discipline — is unchanged, which is a feature. A method your security reviewers already know beats a bespoke framework nobody has practised.
Draw the boundaries first
Before enumerating threats, mark these on the diagram. Most of the interesting findings come out of drawing them accurately rather than out of the STRIDE pass itself:
- Every ingress into a context window. System prompt, user turn, retrieved chunks, tool results, file uploads, transcribed media, other agents’ output. Label each trusted or untrusted; “mostly trusted” is untrusted.
- The provider boundary. Prompts leave your infrastructure. Record where they go, whether they are retained, and under what agreement.
- Every consumer of model output. Renderer, tool dispatcher, database, shell, another model, a queue, a log that somebody later reads in a browser.
- Every credential the agent path can reach, with the identity it acts as. If a tool acts as a service account rather than as the user, mark it — that is a confused-deputy edge.
- Every write into retrieval. Whoever can add a document to the index can put text in a future context.
STRIDE, row by row
| Category | Description |
|---|---|
| Spoofing | The AI-specific version: content spoofing an instruction. Untrusted text impersonates the system prompt or the user's intent. Mitigation is provenance plus privilege separation — never authentication of the text, which is impossible. |
| Tampering | Poisoning the retrieval index, the fine-tuning corpus, a cached response, or the conversation history if it is client-supplied. Mitigation: privileged writes to the index, integrity checks, server-side history. |
| Repudiation | Nobody can reconstruct why the agent did what it did. Sampling makes replay non-deterministic, so the log must capture inputs, model and version, tool calls with arguments, and approvals — not just the final answer. |
| Information disclosure | Data leaving through the answer, through a rendered URL, through a tool argument, through embeddings, or through the provider. Mitigation: entitlement filtering at retrieval, egress allowlists, redaction before ingestion. |
| Denial of service | Both classic exhaustion and its wallet variant: unbounded loops, huge contexts, expensive reasoning modes. Mitigation: quotas, spend caps, loop depth limits, timeouts. |
| Elevation of privilege | The confused deputy. The agent holds authority the requester lacks, and content persuades it to use that authority. Mitigation: act as the user, scope credentials per call, approve irreversible actions. |
A worked fragment
Take a support assistant that answers from a knowledge base and can look up the customer’s order. Three findings fall out of the boundaries almost immediately, and none of them requires imagination:
Elevation. The order-lookup tool takes an order id. The model chooses that id, and the model has read the ticket text, which the customer wrote. Finding: any order id the model can name is an order id it can fetch. Mitigation: the tool derives the account from the authenticated session and filters, so an id outside that account returns not-found regardless of what the model asked for.
Spoofing plus disclosure. Knowledge base articles are editable by support staff, and one is retrieved into a context that also holds order data, in an interface that renders markdown. That is all three legs of the trifecta with an insider or a compromised staff account as the author. Mitigation: strip image URLs from rendered output and allowlist link hosts; separately, review index writes.
Repudiation. A customer claims the assistant promised a refund. The log holds the final message only. Mitigation: log the retrieved chunk ids, model version and tool calls with a trace id, and keep them as long as the dispute window.
What the exercise should produce
Not a document. A diagram with the boundaries marked, and a list of findings each carrying a mitigation, an owner, and a location in the codebase — plus an explicit accepted-risk list, because the alternative is pretending the residual injection rate is zero. Anything you cannot mitigate should be written down as accepted by a named person, which is also what NIST’s AI RMF asks of the MANAGE function.
Re-run it when the capability set changes, not on a calendar. Adding one tool can convert a safe design into a trifecta, and that is the event worth catching. In practice the trigger list is short and worth writing into the pull-request template: a new tool, a new data source in retrieval, a new surface that renders model output, and a change of model or provider. Everything else can wait for the annual pass.