Threat Modeling AI Systems: LLMs, Agents and Adversarial ML

Threat modeling an AI system means doing everything you already do for a web service, then adding three questions classic methods never ask. What can untrusted text make the model decide? What can the model's decisions make your tools do? What can an attacker learn or plant through the model itself? Skip those and your threat model describes the plumbing while missing the part that thinks.

The practical method is short. Draw every path by which data reaches the model and every path by which its output becomes an action. Walk each path with four catalogues: the OWASP Top 10 for LLM Applications 2025, the OWASP Top 10 for Agentic Applications for 2026, MITRE ATLAS and NIST AI 100-2 E2025. Score what you find, and turn the top risks into controls you can test.

The one assumption that should shape every decision: prompt injection is not a bug you will patch. The UK NCSC says current LLMs "simply do not enforce a security boundary between instructions and data". Design so that a fully hijacked model still cannot do serious damage.

Why STRIDE alone misses AI failure modes

STRIDE is still the right starting point. Microsoft's Threat Modeling Tool documentation defines its six categories: spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege. All six still apply to an AI feature.

The gap is that STRIDE assumes components behave as written. A model does not. It follows whichever instructions are most persuasive, including instructions hidden in a retrieved web page or an inbound email. NCSC calls the LLM an "inherently confusable deputy": it holds your privileges and cannot reliably tell your instructions from an attacker's.

Three failure classes fall through a STRIDE-only review:

• Behaviour manipulation through data. A support ticket body is "data" on your data flow diagram. To the model it is a set of instructions. Indirect prompt injection turns every content source into a control channel.
• Attacks on the learning process. Microsoft's AI/ML threat modeling guidance tells teams to assume compromise or poisoning of training data and its provider. It states that training data stores "are part of your Threat Modeling scope". Classic reviews rarely include the data pipeline.
• Leakage through normal use. Model extraction and membership inference use legitimate queries. Nothing is spoofed or tampered with; the attacker just asks well-chosen questions.

Common mistakes: treating the model as a trusted internal component, and threat modeling only the chat endpoint while ignoring the retrieval, memory and tool layers behind it. If you need a refresher on the classic method, the Application Security 101 primer covers it.

Action checklist

• Run a normal STRIDE pass on the AI feature's API, storage and identity first
• Mark the model as an untrusted component on your data flow diagram, not an internal service
• Mark every content source the model reads as a potential instruction channel
• Add the training, fine-tuning and retrieval data pipelines to the review scope
• Write down, in one sentence, what a fully hijacked model could do today

Scope the system: every way data gets in and actions get out

Most AI threat models fail at scoping, not analysis. The diagram shows "user, app, LLM API" and stops. A realistic AI feature has at least eight data flows, and each is an attack path.

Draw these on one diagram:

1. Training data. Usually the provider's, which you cannot inspect. Record the model and version.
2. Fine-tuning data. Yours. Record who can add examples.
3. Retrieval corpus ([RAG](/glossary/rag/)). Documents, tickets and web content pulled into the prompt at runtime.
4. Prompts. System prompt, developer templates and user input. Note what secrets or business rules live in the system prompt.
5. Tools. Every function, API or MCP server the model can call, with the credential each one uses.
6. Outputs. Where model output goes: a browser, a database query, a shell, an email, another agent.
7. Memory. Anything persisted across sessions: summaries, user preferences, embeddings.
8. Model supply chain. Weights, adapters, serving libraries and plugins. NIST AI 100-2 treats supply chain attacks as a distinct generative AI category.

For each flow, record three facts: who can write to it, whose privileges it runs with, and what trust level the content has. A retrieval corpus that any customer can write to has customer-level trust, however internal the database feels.

Then draw the trust boundaries. The one that matters most runs through the prompt itself, where untrusted content sits beside your system instructions. NCSC's guidance is blunt: when an LLM processes a party's information, its privileges should drop to that party's level.

Common mistakes: leaving out the email or ticket bodies the agent reads because "they are just data", and forgetting that model output rendered as HTML or markdown is a code path into the user's browser.

Action checklist

• Draw all eight data flows for your system, even the ones you think are out of scope
• For each flow, record who can write to it and whose privileges it runs with
• List every tool the model can call and the exact credential behind each one
• Mark the trust level of every source that lands in the context window
• Record the model provider, model version and any adapters or plugins in use

Use the threat catalogues together, not one at a time

There is no single list that covers AI systems. Each catalogue answers a different question, and they overlap on purpose. The OWASP agentic list says its entries reference the LLM Top 10 and other OWASP lists rather than replacing them.

The workflow that works: enumerate with the two OWASP lists, express the worst paths as ATLAS technique chains, name ML-layer attacks with NIST's terms, and record ownership and acceptance using the AI RMF functions (Govern, Map, Measure, Manage). The Agent Security Map threat index mirrors both OWASP lists with one page per ID if you want a quick reference.

NIST's taxonomy splits attacks by attacker objective: availability, integrity and privacy, plus misuse enablement for generative AI (getting the model to produce what its developers tried to restrict).

Common mistakes: using only the LLM list for an agent, which misses delegation and cascading failures, and copying catalogue entries into a spreadsheet without tying each one to a specific data flow on your diagram.

Action checklist

• Walk every data flow against the ten OWASP LLM 2025 entries
• If the system can call tools or act autonomously, walk it against ASI01 to ASI10 as well
• Express your top three attack paths as sequences of MITRE ATLAS techniques
• Use NIST AI 100-2 terms (evasion, poisoning, privacy, misuse) in the threat register
• Assign each risk an owner using the AI RMF Govern function

Adversarial ML threats: evasion, poisoning, extraction, inference

These are attacks on the model itself. They matter most when you train or fine-tune, or when a model makes a security decision such as fraud scoring. If you call a hosted LLM, the provider carries most training-time risk; you still own fine-tuning data and your API.

Evasion. The attacker crafts inputs that the model misclassifies at inference time. NIST lists evasion under integrity violations, and ATLAS tracks it as Evade AI Model (AML.T0015). Microsoft's guidance gives the canonical example: forcing malicious email past a spam classifier. Mitigations include adversarial training, input validation against a defined well-formed input, and slowing the query loop attackers use to search for a working perturbation.

Data poisoning. The attacker contaminates training or fine-tuning data so the model learns the wrong behaviour, sometimes with a hidden trigger (a backdoor). Microsoft calls data poisoning "the greatest security threat in machine learning today" for lack of standard detections. ATLAS calls it Training Data Poisoning (AML.T0020). Mitigations: provenance and lineage tracking, anomaly detection on data distribution, and the ability to roll back to a prior model version.

Model extraction. The attacker rebuilds a functional copy of your model through legitimate queries. An extracted model lets them develop evasion attacks offline. ATLAS lists Extract AI Model (AML.T0024.002). Microsoft recommends returning only the minimum detail callers need, such as rounded confidence values.

Membership inference. The attacker determines whether a specific record was in the training set. That is a privacy breach even if no record is ever displayed. ATLAS lists Infer Training Data Membership (AML.T0024.000), and Microsoft points to differential privacy as the research-backed mitigation.

Common mistakes: assuming these attacks are academic because they need ML skill, and exposing raw confidence scores or log probabilities in an API because a developer found them useful for debugging.

Action checklist

• Record whether you train or fine-tune, and who can contribute data to that pipeline
• Track provenance for every fine-tuning and retrieval dataset you control
• Confirm you can roll back to a known-good model version within a defined time
• Remove raw confidence scores and log probabilities from external API responses
• Rate-limit and monitor query volume on any model endpoint that makes a security decision

LLM threats: injection, disclosure, output handling, excessive agency

Four entries from the OWASP 2025 list deserve the most attention in LLM features.

Prompt injection ([LLM01:2025](https://guptadeepak.com/agent-security/threats/llm01/)). OWASP separates direct injection, where the user's own input alters behaviour, from indirect injection, where the model "accepts input from external sources, such as websites or files". OWASP itself says "it is unclear if there are fool-proof methods of prevention". The real-world reference case is EchoLeak, listed by NVD as CVE-2025-32711. NVD describes it as AI command injection in M365 Copilot allowing information disclosure, and Microsoft scored it 9.3. The OWASP agentic list cites it as a zero-click attack triggered by a single crafted email. The prompt injection glossary entry covers the basics.

Sensitive information disclosure ([LLM02:2025](https://guptadeepak.com/agent-security/threats/llm02/)). The model reveals data from its context, its retrieval corpus or connected systems. In RAG systems, check whether retrieval respects the requesting user's permissions. LLM08:2025 (Vector and Embedding Weaknesses) covers retrieval stores that leak across permission boundaries.

Improper output handling ([LLM05:2025](https://guptadeepak.com/agent-security/threats/llm05/)). Downstream systems trust model output. Output rendered as HTML is cross-site scripting. Output passed to SQL or a shell is injection. Treat model output exactly like user input, because an attacker may have written it.

Excessive agency ([LLM06:2025](https://guptadeepak.com/agent-security/threats/llm06/)). OWASP names three root causes: excessive functionality, permissions and autonomy. Its mitigations are the ones to copy into your design: minimise extensions, require user approval for high-impact actions, enforce authorization in downstream systems ("complete mediation"), and run actions in the user's own context.

The design rule that ties these together comes from NCSC. Do not try to filter every malicious phrase; attackers rephrase. Constrain what the system can do, so a successful injection hits a wall.

Common mistakes: relying on a system prompt instruction like "never reveal customer data" as a control, putting secrets in the system prompt (LLM07:2025), and building a deny-list of known jailbreak strings.

Action checklist

• List every source of indirect input (email, tickets, web pages, files) and the tools reachable from it
• Confirm retrieval filters documents by the requesting user's permissions before the model sees them
• Encode or sanitise all model output before it reaches a browser, query, shell or template
• Remove any secret, credential or sensitive rule from the system prompt
• Enforce authorization in the downstream API, never in the prompt
• Require human approval for any irreversible or externally visible action

Agentic threats: tools, privilege, memory, delegation, cascades

Agents raise the stakes because the model's output becomes an action. NIST AI 100-2 notes that hijacked agents can be made to execute arbitrary code or exfiltrate data from their environment. The OWASP agentic list adds a principle worth putting in your design review: least agency. Do not grant autonomy the task does not need, because unneeded autonomy expands the attack surface without adding value.

The five agentic threats to model explicitly:

• Tool misuse ([ASI02](https://guptadeepak.com/agent-security/threats/asi02/)). The agent calls a legitimate tool in a harmful way, such as sending an email to an attacker-supplied address. ATLAS tracks this as AI Agent Tool Invocation (AML.T0053) and Exfiltration via AI Agent Tool Invocation (AML.T0086). Constrain parameters, not just tool access. For MCP-based tools, follow Secure an MCP Server.
• Identity and privilege abuse ([ASI03](https://guptadeepak.com/agent-security/threats/asi03/)). The agent runs on a broad service account or inherits a user's full session. Give each agent its own identity and short-lived, task-scoped credentials. Identity for AI Agents covers the build, and the agent identity overview maps the vendor primitives.
• Memory and context poisoning ([ASI06](https://guptadeepak.com/agent-security/threats/asi06/)). An attacker writes instructions into persistent memory, and they fire in a later session. ATLAS models this as AI Agent Context Poisoning, sub-technique Memory (AML.T0080.000). Validate memory writes, record their source, and expire them.
• Delegation and inter-agent trust ([ASI07](https://guptadeepak.com/agent-security/threats/asi07/)). One agent accepts another's output as authoritative. Authenticate agent-to-agent messages and re-check authorization at every hop rather than passing a parent's privileges down.
• Cascading failures ([ASI08](https://guptadeepak.com/agent-security/threats/asi08/)). One poisoned or wrong output propagates through a chain of agents and gets amplified. OWASP frames ASI08 around propagation rather than origin. Add circuit breakers, rate limits on actions, and a kill switch that works across the whole chain.

Also model ASI01 (Agent Goal Hijack), the agentic form of injection, and ASI09 (Human-Agent Trust Exploitation). Approval prompts only help if the human can see what they are approving.

Network controls are your last line. If a hijacked agent cannot reach an attacker's server, most exfiltration fails. Egress Control for AI Agents shows how to build that allowlist.

Common mistakes: giving the agent the developer's own API token during a pilot and never replacing it, and letting approval prompts summarise an action in the model's own words instead of showing the exact parameters.

Action checklist

• For each tool, record allowed parameters, not just whether the agent can call it
• Give each agent its own identity with short-lived, task-scoped credentials
• Record the source of every memory write and set an expiry on stored memory
• Re-check authorization at every agent-to-agent hop
• Build a kill switch that stops the whole agent chain, and test it
• Show exact action parameters in every human approval prompt
• Restrict agent network egress to an allowlist of required destinations

Worked example: a support agent with ticketing and email tools

Here is a compact threat model for a common design. The agent reads inbound support tickets, searches a help-centre corpus (RAG), drafts replies, updates ticket fields, and can send email to the customer. It stores a short summary of each customer's history as memory. It runs with a service account that has read and write access to the ticketing system.

The key observation: the ticket body is written by anyone on the internet, and it lands in the same context as the agent's instructions. Every customer is therefore a potential prompt author for an agent holding a privileged service account.

Scoring uses a simple 1 to 5 scale for likelihood and impact, multiplied. The scores below are illustrative judgments for this design, not measured data. Calibrate them to your own environment.

Three design changes remove most of the risk in this table:

1. Take the recipient away from the model. The email tool accepts a ticket ID and a body. Code looks up the recipient. Threat 1 drops from catastrophic to "the agent can send a strange reply to the person who wrote the ticket".
2. Replace the service account with a per-ticket token. Threat 2 now hits the ticketing API's own authorization check.
3. Require approval for anything outside a reply. Refunds, account changes and escalations go to a human queue that shows exact parameters.

A better system prompt and an injection classifier are absent from the primary mitigations on purpose. Add both as defence in depth, but neither should carry a score-20 risk, because both fail when the attacker finds new wording.

Common mistakes: scoring likelihood low because "our customers are not attackers", and stopping at mitigations without recording the residual score after the control is in place.

Action checklist

• Build a risk table like this one for your own agent, with catalogue IDs on every row
• For each score of 15 or higher, name a mitigation that does not depend on model behaviour
• Move recipient, target account and amount parameters out of the model and into code
• Record the residual score after each mitigation, and who accepted it
• Re-score the table whenever a new tool or data source is added

Turn the model into controls, tests and ongoing review

A threat model that does not change the build is a document, not a control. Convert each high-scoring row into a design control, a test that proves it, and an owner.

Controls. Group them by layer so nothing falls between teams:

• Identity: per-agent identity, scoped tokens, no shared service accounts. See Replace Static API Keys for the credential side.
• Tool layer: parameter constraints, approval gates, authorization enforced downstream.
• Data layer: permission-aware retrieval, provenance on corpora, memory validation.
• Runtime: input and output filtering as defence in depth. The runtime guardrails overview and the guardrails glossary entry cover the category.
• Network: egress allowlists for every agent runtime.
• Logging: inputs, outputs and every tool call with its parameters. NCSC recommends this logging specifically to detect injection attempts.

Tests. Every mitigation needs an adversarial test that would fail if the control broke. Turn each risk row into test cases: direct and indirect injection payloads aimed at that tool, memory poisoning attempts, and cross-tenant retrieval probes. Red-Team an LLM covers building the probe set and scoring results. NIST AI 100-2 lists open-source tools such as Garak and PyRIT, and AgentDojo for agent injection testing. If you are choosing a commercial product, compare categories on the agentic AI security tools page.

Ongoing review. AI systems change without a code deploy. Re-run the threat model when:

• A new tool, MCP server or data source is connected
• The model or model version changes
• The agent gains autonomy, such as removing an approval step
• A new entry appears in the OWASP lists or a relevant ATLAS case study is published
• A red team finding or production incident shows a missed path

For governance, the AI RMF's Govern and Manage functions are where risk acceptance belongs. The CISO checklist for agent security gives leadership a shorter version of the same questions, and Verify an AI Agent covers checking third-party agents before you connect them.

Common mistakes: running the threat model once before launch and never again, and writing tests that check the model refuses rather than tests that check the tool layer blocks the action.

Action checklist

• For every risk scored 12 or higher, record a control, a test and a named owner
• Add adversarial test cases for each high risk to your CI or release gate
• Log every tool call with its full parameters and the identity that made it
• Define the triggers that force a threat model review, and put them in your change process
• Re-run the full model at least once per model version change

Your complete checklist

• Run a STRIDE pass first, then treat the model as an untrusted component
• Draw all eight data flows: training, fine-tuning, retrieval, prompts, tools, outputs, memory, supply chain
• Record who can write to each flow and whose privileges it runs with
• Walk every flow against OWASP LLM 2025, and against ASI01 to ASI10 if the system uses tools
• Track provenance for every dataset you train, fine-tune or retrieve from
• Filter retrieval by the requesting user's permissions before the model sees content
• Encode all model output before it reaches a browser, query, shell or template
• Keep secrets and sensitive rules out of system prompts
• Enforce authorization in downstream APIs, never in the prompt
• Move recipients, target accounts and amounts out of the model and into code
• Give each agent its own identity with short-lived, task-scoped credentials
• Validate, attribute and expire every memory write
• Require human approval with exact parameters for irreversible actions
• Log every tool call with parameters and identity
• Build a risk table with residual scores and named owners
• Turn each high risk into an adversarial test in your release gate
• Re-run the threat model on every new tool, data source or model version