The AI Security Landscape — OWASP Top 10 for LLM Applications

Why AI systems create new attack surfaces, the OWASP Top 10 for LLM applications, and MITRE ATLAS as a map of real attacks.

Est. 35 min Updated Suggest a fix
// Part 01 — The big picture

AI Adds New Trust Boundaries; It Does Not Replace Old Security Work

An LLM feature is a software system that accepts data, calls a model, and often returns an answer or triggers another operation. The model introduces a new kind of uncertainty: it produces probabilistic text from both trusted instructions and untrusted context. That does not make every model call a vulnerability, but it does mean a prompt cannot serve as an access-control boundary.

Think of a campus help desk that answers questions from public policy pages and private student records. The feature has ordinary web risks (authentication bugs, exposed credentials, unsafe output) plus AI-specific risks such as instruction conflict, retrieval leakage, and an agent taking an action it should not take. Security belongs in the surrounding application: identity checks, data permissions, narrow tools, validation, monitoring, and tested failure paths.

💡 Note
An instruction such as "never reveal a private record" is useful behavior guidance, not an authorization mechanism. The database and application must enforce who can read each record.
// Part 02 — Risk taxonomy

Read the OWASP Top 10 as a Starting Map

The OWASP Top 10 for Large Language Model Applications (2025 edition) groups recurring security concerns into ten categories. It is an awareness and risk-identification resource, not a severity ranking for your particular product and not a checklist that proves a system is secure. Use the names consistently, then connect each concern to your own data flows, permissions, and consequences.

2025 categoryWhat to ask in your design review
LLM01 Prompt InjectionCan user or retrieved content influence behavior beyond the intended task?
LLM02 Sensitive Information DisclosureCould the model or application expose information a user is not allowed to see?
LLM03 Supply ChainCan dependencies, models, datasets, or hosted services change or be tampered with?
LLM04 Data and Model PoisoningCould manipulated training, fine-tuning, or retrieval data change system behavior?
LLM05 Improper Output HandlingIs generated content safely validated before rendering or passing to another system?
LLM06 Excessive AgencyDoes the application grant more tools, permissions, or autonomy than the task needs?
LLM07 System Prompt LeakageCould internal instructions or sensitive prompt contents be exposed?
LLM08 Vector and Embedding WeaknessesCan vector creation, storage, or retrieval cause manipulation or data exposure?
LLM09 MisinformationCould users rely on plausible but inaccurate output without suitable checks?
LLM10 Unbounded ConsumptionCan requests or loops consume excessive model, compute, or service resources?
🎯 Pro Tip
Prioritise by product context. A study assistant may focus on student-record confidentiality and misinformation; an internal coding agent may place greater emphasis on tool permissions, output handling, and supply-chain controls.
// Part 03 — Attack surface

Draw the Whole Request Path

Start from a user action and follow every boundary: browser, application server, identity provider, prompt builder, model provider, retrieval index, databases, tools, logs, and human reviewer. Include data that arrives indirectly—an uploaded PDF, a support ticket, a web page fetched by a tool, or a code repository—not just the prompt box. A source can be legitimate business data and still be untrusted as an instruction.

  1. List assets: credentials, personal data, tenant records, model configuration, indexes, and actions that affect users or money.
  2. Name actors and entry points: signed-in users, administrators, external content, scheduled jobs, and service identities.
  3. Mark trust boundaries and data flows, including provider calls, logs, queues, and agent tool results.
  4. For each flow, ask what could go wrong, what impact follows, and which control is enforced outside the model.
  5. Record an owner and a test for each important risk; revisit the model when a tool, data source, or permission changes.

For an Indian delivery-support assistant, order status and phone numbers are sensitive assets; an order lookup is a privileged action; the support transcript and a retrieved FAQ have different trust levels. These distinctions guide what can enter the prompt and what the assistant may do.

// Part 04 — MITRE ATLAS

Use ATLAS to Learn Adversary Behavior

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a living knowledge base of adversary tactics and techniques involving AI-enabled systems. It helps security teams discuss observed or plausible adversary behavior and connect threat scenarios to mitigations. The knowledge base evolves, so consult its current matrix and technique pages rather than copying an old technique count or assuming an entry applies to every model.

OWASP and ATLAS serve different but complementary purposes. OWASP’s 2025 list is a compact set of application risk categories for awareness and design conversations. ATLAS organizes adversary tactics, techniques, and case studies in a matrix. Neither is a certification, and neither replaces your asset inventory, architecture review, or application-specific tests.

ResourceUse it forDo not use it as
OWASP Top 10 for LLM ApplicationsShared vocabulary for common LLM application risk areasA complete threat model or a guarantee of security
MITRE ATLASExploring adversary tactics and techniques and identifying relevant mitigationsA prediction that every technique will affect your system
Your threat modelPrioritising risks for your architecture, users, data, and impactA one-time document that never changes
// Part 05 — Hands-on

Build a Small Risk Register

A useful first artifact is a short register that connects a scenario to an asset, impact, control owner, and evidence. The following Python example is self-contained and deliberately uses fictional records. It helps structure a review; it does not calculate risk automatically or substitute for security judgement.

risk_register.py
python
from dataclasses import dataclass

@dataclass(frozen=True)
class Risk:
    scenario: str
    asset: str
    impact: str
    control: str
    evidence: str

risks = [
    Risk(
        scenario="A user receives another tenant's retrieved support note",
        asset="Customer support records",
        impact="Privacy breach and loss of trust",
        control="Filter retrieval by authenticated tenant before context assembly",
        evidence="Automated cross-tenant access tests",
    ),
    Risk(
        scenario="A generated summary is rendered as executable markup",
        asset="Support agent browser session",
        impact="Unsafe browser behavior",
        control="Render as text or use a maintained output sanitizer",
        evidence="Output-encoding tests in the web application",
    ),
]

for risk in risks:
    print(f"Scenario: {risk.scenario}")
    print(f"  Asset: {risk.asset}; impact: {risk.impact}")
    print(f"  Control: {risk.control}")
    print(f"  Evidence: {risk.evidence}")

A register is only as good as its evidence. Replace vague entries such as "the prompt says not to" with a control enforced at the database, API, identity, sandbox, or human approval boundary and an observable test result.

📌 Real World Example
A benign test can verify that a user assigned to Tenant A gets no Tenant B documents. The expected result is an empty or appropriately filtered retrieval—not a carefully crafted prompt that hopes the model hides the document.
// Part 06 — Production

Turn Categories into Controls and Tests

For each risk, separate prevention, detection, and recovery. Prevention might mean a database predicate bound to the authenticated identity. Detection might mean an audit event for a denied tool call. Recovery might mean revoking a credential, disabling a tool integration, or notifying affected users under your incident process. Measure controls at the layer that owns them rather than scoring the model’s promise to comply.

  1. Map each feature to the OWASP categories that matter; document why a category is or is not relevant.
  2. Use ATLAS to enrich scenarios with adversary behavior and mitigation ideas, then adapt them to your architecture.
  3. Write acceptance tests for authorization, unsafe output, resource limits, and tool boundaries.
  4. Keep harmless regression examples for prompt and retrieval behavior; avoid placing real secrets in tests or logs.
  5. Assign review ownership and repeat the exercise after material changes to models, tools, data, or providers.

Treat model output as untrusted input even when it is fluent. Encode it for its destination, validate structured fields against a schema, enforce access in the service layer, and cap time, tokens, retries, and tool calls. These controls address different failure modes; no single filter covers them all.

// Part 07 — Pitfalls

Common Mistakes

Treating the OWASP list as a ranked checklist

Why: A general taxonomy cannot know your product assets or impact.

Fix: Build a system-specific threat model and prioritize by realistic impact and exposure.

Treating prompt wording as an authorization control

Why: Model instructions are not enforced access checks and may be overridden or misunderstood.

Fix: Authorize every read and action in deterministic application code.

Assuming ATLAS predicts a specific attack

Why: A knowledge base describes behaviors and mitigations, not certainty about your deployment.

Fix: Use relevant entries to create testable scenarios and record your assumptions.

Reviewing only the model call

Why: Data import, retrieval, browser rendering, logs, and downstream tools are also part of the system.

Fix: Trace end-to-end flows and include every trust boundary in the review.

// Part 08 — Practice

Map a Student Support Assistant

Practice

Create a first-pass AI security map

  1. Draw a data-flow diagram for a student assistant that answers from public FAQs and private account records.
  2. List at least five assets and identify which user or service identity may access each one.
  3. Select relevant OWASP Top 10 for LLM Applications (2025) categories and explain the architecture-specific reason for each.
  4. Find one relevant current MITRE ATLAS scenario or mitigation, and record its official page title and why it applies.
  5. Write four risk-register entries with a concrete control owner and a testable piece of evidence.

Hint: Start with trust boundaries: browser to application, application to identity/database, application to model provider, and retrieved content to prompt. Keep test data fictional.

🎯 Key Takeaways

  • ✓LLM applications inherit ordinary software risks and add uncertainty around instructions, generated output, and connected tools.
  • ✓The OWASP Top 10 for LLM Applications 2025 provides ten shared risk categories, not a product-specific ranking or security guarantee.
  • ✓MITRE ATLAS is a living knowledge base for adversary tactics and techniques involving AI systems; use its current entries as scenario input.
  • ✓A prompt is not an access-control boundary; enforce permissions in the application, data store, and tool APIs.
  • ✓Threat models should trace assets, identities, trust boundaries, data flows, controls, and test evidence end to end.
  • ✓Revisit the model when the system changes and assign owners to prevention, detection, and recovery controls.
  • Official references: OWASP Top 10 for LLM Applications (consult the 2025 edition and current project pages).
  • Official reference: MITRE ATLAS, a living knowledge base; review the current matrix, technique pages, and mitigations.

Next

What comes next

You have a vocabulary for AI security risks. Next, map assets, trust boundaries, and data flows, then turn likely threats into controls and tests.

Lesson 02 · Threat Modelling AI Systems →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...