Threat Modelling AI Systems
Map data flows, trust boundaries and assets in an LLM app, then apply STRIDE-style analysis to prompts, retrieval, tools and outputs.
Threat Modelling Is a Design Conversation
Threat modelling is a structured way to ask what a system protects, who can interact with it, where trust changes, and what could fail. It is not a prediction that every scenario will happen. A useful model exposes assumptions while architecture can still change, then points to a control or test that a team can own.
For an LLM application, do not draw only a box labeled "model." Include identity, prompt assembly, retrieval, model provider, tools, result rendering, logs, and human review. User text and retrieved content are inputs, not trusted policy. A model output that will affect a database or user account crosses a particularly important boundary.
Name Assets, Actors and Security Objectives
Begin with a narrow feature and concrete assets. For a support assistant, assets might include customer identifiers, account details, support transcripts, provider credentials, and the ability to issue an account credit. Actors include end users, support staff, administrators, the application service, the model provider, and external content sources. Give service identities names too; they often have more access than a person.
| Question | Example for a support assistant |
|---|---|
| What must be protected? | Order details, account data, provider secrets, and payment adjustments |
| Who may use it? | Authenticated customer, support agent, administrator, background job |
| What may each actor do? | Read their own status; staff may view assigned cases; only authorized staff may approve credits |
| What is the impact of failure? | Cross-customer disclosure, unauthorized change, service disruption, or incorrect advice |
| What is explicitly out of scope? | Model internals the application cannot observe; record the uncertainty instead of assuming safety |
Include privacy and integrity alongside availability. In a learning product, a wrong answer can be harmful without exposing data; in a commerce workflow, a seemingly small action may affect payment or delivery. State the consequence in the language of the product so engineering and operations can prioritize it.
Draw Boundaries Before Listing Attacks
Use a simple data-flow diagram. Represent people and external services as actors, application functions as processes, data stores as stores, and arrows as flows. Mark where identity is checked, where data changes trust level, and which service account performs a request. A retrieval index is a data store; it is not an authorization layer by itself.
- Choose one user journey and mark its start, result, and failure paths.
- Draw the browser, application API, identity service, database, index, model provider, and every tool that can be called.
- Label each arrow with data type, identity, and purpose; note sensitive fields and whether content is user-controlled.
- Draw a trust boundary around each component operated by a different team or organization.
- Mark enforcement points: authorization checks, schema validation, output encoding, rate limits, and approvals.
Ask What Can Go Wrong at Each Boundary
STRIDE is a classic threat-modeling mnemonic: Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, and Elevation of privilege. Applying it to AI systems is a useful analysis technique, not a special OWASP or MITRE scoring method. Some threats overlap, and an AI-specific scenario may fit more than one category.
| STRIDE lens | AI application example | Control direction |
|---|---|---|
| Spoofing | A request is attributed to the wrong user or service identity | Verify identity at the API and bind requests to authenticated principals |
| Tampering | An imported document or model response changes data used downstream | Track provenance; validate content and enforce integrity at each write boundary |
| Repudiation | A privileged AI-assisted action has no attributable record | Audit actor, authorization result, approval, and action with privacy-aware retention |
| Information disclosure | Retrieved context crosses a tenant or role boundary | Enforce access before context assembly and re-check sensitive reads |
| Denial of service | Repeated prompts or agent loops exhaust model or service capacity | Apply quotas, timeouts, budgets, bounded retries, and concurrency controls |
| Elevation of privilege | The assistant invokes a tool beyond the user’s permission | Use narrow server-side tools and authorize each operation independently |
Represent a Boundary and Its Checks
The small Python model below describes a hypothetical flow and verifies that sensitive operations have an enforcement point. It is plain data you can run as-is, not a threat-scoring engine. Extend it with your own services and have reviewers challenge whether the listed controls actually exist in code and deployment.
from dataclasses import dataclass
@dataclass(frozen=True)
class Flow:
source: str
destination: str
data: str
trust: str
enforcement: str
flows = [
Flow("browser", "application API", "question and session", "untrusted to trusted", "authenticate request"),
Flow("application API", "retrieval store", "tenant-scoped query", "trusted service to data", "check tenant and document ACL"),
Flow("application API", "model provider", "minimized prompt context", "trusted to external service", "redact and constrain output"),
Flow("application API", "credit tool", "customer ID and proposed amount", "assistant proposal to privileged action", "authorize and require approval"),
]
for flow in flows:
print(f"{flow.source} -> {flow.destination}: {flow.data}")
print(f" Trust: {flow.trust}; enforcement: {flow.enforcement}")
if "privileged" in flow.trust and not flow.enforcement:
raise ValueError("Privileged flow has no enforcement point")A real diagram should distinguish the model’s suggestion from the application’s decision. The model may propose an action, but the service should parse a narrow schema, identify the human or service principal, re-authorize the operation, and record the result. A control on the diagram is a hypothesis until a test demonstrates it.
from dataclasses import dataclass
@dataclass(frozen=True)
class Scenario:
asset: str
precondition: str
threat: str
impact: str
test: str
case = Scenario(
asset="Customer order records",
precondition="A support user can ask the assistant about an order",
threat="The response includes a record outside the user's allowed scope",
impact="Private information is disclosed to another customer",
test="As Tenant A, request a Tenant B fixture and assert no B fields appear in retrieved context",
)
print(case)Choose Controls by Impact and Evidence
A practical review can record likelihood, impact, existing control, and evidence without pretending the numbers are precise. Use a consistent qualitative scale if it helps teams discuss priorities, but keep the assumptions visible. Prioritize risks that combine sensitive assets, broad access, weak boundaries, and difficult recovery.
- Describe a concrete precondition and undesired outcome; avoid generic labels such as “AI is unsafe.”
- Identify the system component that must enforce a control and the team accountable for it.
- Specify a positive test and a negative test, including expected behavior when a dependency fails.
- Record residual risk and uncertainty, especially for third-party services and probabilistic model behavior.
- Revisit findings when permissions, tools, retrieval sources, model providers, or user workflows change.
Common Mistakes
Drawing only the model provider
Why: Important risks often occur in prompt construction, retrieval, tools, logs, and output rendering.
Fix: Trace one complete user journey through every service and store.
Treating a vector database as an access-control policy
Why: Similarity search ranks content; it does not establish a user’s authorization.
Fix: Bind retrieval to the authenticated principal and enforce ACLs before adding content to context.
Using STRIDE as a numeric risk score
Why: STRIDE names threat categories; it does not tell you likelihood or business impact.
Fix: Use it to prompt analysis, then separately document impact, likelihood assumptions, and evidence.
Listing controls without tests
Why: A design statement can drift from implementation or fail under a dependency outage.
Fix: Attach a named owner and automated or operational evidence to each important control.
Threat-Model a Retrieval Assistant
Practice
From architecture sketch to test plan
- Choose an assistant that searches private account records and public help content; define its users and one privileged action.
- Draw a data-flow diagram with at least six components, trust boundaries, and the identity on each service-to-service flow.
- Write one threat scenario for each STRIDE category and identify any categories that overlap.
- For the two highest-impact scenarios, write a prevention control, a negative test, and an incident signal.
- Add a failure case for unavailable identity, retrieval, or model provider services and describe the safe behavior.
Hint: Use fictional tenant data. A strong scenario states precondition, attacker capability, asset, impact, and observable evidence—not just an attack name.
🎯 Key Takeaways
- ✓Threat modelling starts with assets, actors, objectives, and one complete user journey.
- ✓Draw the model provider, prompt builder, retrieval, tools, stores, rendering, logs, and service identities as part of the system.
- ✓STRIDE is a useful classic mnemonic when adapted to AI data flows; it is not a severity score.
- ✓The model can propose content or actions, but application services must enforce identity, authorization, validation, and limits.
- ✓Every important risk needs a control owner, a testable outcome, and documented residual uncertainty.
- ✓Revisit the threat model after meaningful architecture, permission, provider, or workflow changes.
- Reference: Microsoft’s STRIDE threat-modeling methodology is a classic framework; apply it as an analytical aid, not as an AI-specific standard.
- Reference: OWASP Top 10 for LLM Applications and the current MITRE ATLAS knowledge base can inform scenarios and mitigations.
Next
What comes next
You have mapped boundaries and controls. Next, examine direct and indirect prompt injection, why instruction filters are incomplete, and how to reduce the impact.
Lesson 03 · Prompt Injection and Jailbreaks — Direct and Indirect →HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
