AI and LLM Security Basics — Prompt Injection, Data Leakage, Model Abuse

A concise security orientation to prompt injection, sensitive-data exposure, and model abuse, with practical controls and a path to the full AI Security course.

Est. 15 min Updated Suggest a fix

A Short Orientation to AI Security

Large language models (LLMs) turn user input and surrounding context into generated output. They can summarize a support ticket, search internal documents, or call a tool. That flexibility introduces security risks at the boundaries between untrusted content, confidential data, model instructions, and actions. This lesson is an introduction, not a complete AI security curriculum.

Prerequisites: basic familiarity with web applications, APIs, authentication, and the confidentiality-integrity-availability model. Objective: identify three common LLM risk families and choose controls that keep model output from becoming trusted authority. Work only with systems and data you own or are authorised to assess.

A simple AI feature data flow

User request ──► application ──► model provider
                    │                 │
                    ├── retrieved documents
                    ├── tools / APIs ◄─┘
                    └── logs, traces, and stored conversation state

Security review: what crosses each boundary, who can influence it,
and which component is authorised to take an action?

The model is a probabilistic component, not an access-control engine. A system that gives the model broad data access or powerful tools must apply conventional security controls around those capabilities.

Prompt Injection: Instructions Are Not a Security Boundary

Prompt injection occurs when untrusted text influences a model to depart from the intended task. The text may arrive directly in a user prompt or indirectly inside a web page, email, document, code comment, or retrieved passage. A model sees a combined context; merely labelling one part as system instructions and another as data does not create a hard isolation boundary.

Threats and the control boundary

RiskExampleControl that must exist outside the model
Prompt injectionA retrieved page tries to redirect the assistantKeep retrieval scoped; treat retrieved text as untrusted input
Data leakageA reply includes another learner's recordAuthorize retrieval for the current identity before model context is built
Model abuseA user floods a costly generation endpointUse identity-aware quotas, limits, monitoring, and an abuse response path

Direct and indirect paths

Direct: a user asks the assistant to ignore its task and reveal protected data.
Indirect: a retrieved document contains adversarial instructions that the model
          treats as part of the task while summarising the document.

Risk increases when the model can also:
  • retrieve records that the caller is not allowed to see
  • send email, update a ticket, execute a workflow, or call a payment API
  • write persistent memory that will affect later users or sessions

The main concern is not a particular string. It is a trust-boundary failure: untrusted content can influence an action or disclosure. Text pattern filters can be bypassed and may block legitimate requests, so they should not be the primary defense.

⚠️ Important
Never evaluate a prompt-injection test against a production assistant that has access to real customer records, secrets, or external tools. Use a disposable test environment with synthetic data and mocked actions.

Keep Retrieval and Tool Use Inside the User's Permissions

Retrieval-augmented generation (RAG) adds documents to the model's context. Apply the same authorisation rules to retrieval that the rest of the application uses. Filter by the authenticated user's permissions before content reaches the model; do not rely on the model to decide which document is appropriate. Treat retrieved text as untrusted input even when it came from an internal system.

Tools need their own server-side authorization, validation, and audit trail. The model may suggest a tool call, but the application should decide whether the authenticated user may perform that action, validate its arguments, and require confirmation for consequential operations. Prefer narrowly scoped tools over a general-purpose shell or unrestricted database connection.

Safer tool boundary

1. Authenticate the person making the request.
2. Authorise that person for the exact record and operation.
3. Ask the model for a structured proposal, not an executable command.
4. Validate the proposal against a schema and business rules.
5. Require human confirmation for high-impact or irreversible actions.
6. Execute through a least-privilege service and record the result.

The model can propose. Trusted application code authorises and executes.

A delivery-support assistant might draft a refund recommendation. The order service—not the model—must verify order ownership, refund eligibility, amount limits, and approval requirements before changing payment state.

Data Leakage: Minimise Before Sending or Logging

An LLM feature can expose sensitive information through prompts sent to a provider, model replies, retrieval results, browser-visible output, traces, or support logs. Data minimisation is stronger than asking a model to keep secrets. Send only fields needed for the task, redact identifiers where practical, and confirm provider, retention, access, and contractual settings before processing regulated or confidential information.

Build a simple redaction boundary

import re

EMAIL = re.compile(r"[w.+-]+@[w.-]+.[A-Za-z]{2,}")


def redact_email(text: str) -> str:
    """Illustrative minimisation; not a complete DLP solution."""
    return EMAIL.sub("[REDACTED_EMAIL]", text)


support_note = "Contact learner@example.org about the course refund."
model_input = redact_email(support_note)
print(model_input)
# Contact [REDACTED_EMAIL] about the course refund.

This example demonstrates one narrow transformation, not a universal detector. It will not identify every personal identifier, secret, or context-sensitive disclosure. Use a maintained data-loss-prevention approach, test it with representative synthetic cases, and keep authorization checks independent of redaction.

🎯 Pro Tip
Keep secrets out of prompts and model-visible context entirely when the task can be completed without them. Redaction is a fallback control, not permission to collect more data.

Model Abuse and Unsafe Output

Abuse includes attempts to use a feature for spam, fraud, harassment, prohibited content, or resource exhaustion. The risk depends on the product: a public text generator, an internal summarizer, and an assistant that can initiate account changes need different controls. Apply identity-aware quotas, request-size limits, timeouts, cost monitoring, and a clear abuse-reporting path.

Treat model output as untrusted data. HTML-escape text before rendering; parameterise database queries; validate structured output against a strict schema; and never execute generated code or shell commands automatically. A refusal or safety classifier can help, but it does not replace authorization, output encoding, or a human review for high-impact decisions.

import json
from typing import Any


def parse_refund_proposal(raw: str) -> dict[str, Any]:
    """Parse a model proposal; a real service must also authorise and review it."""
    value = json.loads(raw)
    if not isinstance(value, dict):
        raise ValueError("proposal must be an object")
    if set(value) != {"order_id", "amount_paise", "reason"}:
        raise ValueError("unexpected fields")
    if not isinstance(value["order_id"], str) or not value["order_id"].strip():
        raise ValueError("order_id is required")
    if type(value["amount_paise"]) is not int or value["amount_paise"] < 0:
        raise ValueError("amount must be a non-negative integer")
    if not isinstance(value["reason"], str) or len(value["reason"]) > 300:
        raise ValueError("reason is invalid")
    return value


proposal = parse_refund_proposal(
    '{"order_id":"demo-123","amount_paise":5000,"reason":"Duplicate charge"}'
)
print(proposal["order_id"])

Schema validation confirms shape, not truth or entitlement. The order identifier still needs to be looked up under the caller's identity, the amount checked against the order and policy, and the action approved through the ordinary service workflow.

A Lightweight Threat-Model Checklist

Before launch or a significant model change, document the feature, data, and consequences—not just the prompt text.

Feature: What user goal does the model support?
Assets: Which personal, financial, business, or authentication data is involved?
Inputs: Can users, retrieved documents, or third parties influence the context?
Access: Are retrieval and tools scoped to the current identity and tenant?
Outputs: Can generated content trigger code, a database write, or a decision?
Controls: What limits, review steps, audit events, and rollback paths exist?
Testing: Are synthetic cases covering injection, leakage, malformed output, and abuse?
Operations: Who reviews alerts, handles reports, and updates the risk assessment?

Repeat the review when a model gains new tools, a new document source is connected, or generated output begins affecting a higher-impact decision. A prompt change alone may not be the largest change in the system's risk.

Common Mistakes

Common mistakes

Mistake: “The system prompt says not to reveal data, so it is safe.”
Why: A model instruction is not an access-control boundary.
Fix: Enforce record-level authorization before retrieval and tool execution.

Mistake: “We block a list of suspicious prompt phrases.”
Why: Meaning can be expressed in many forms; static phrase lists are brittle.
Fix: Reduce capabilities, isolate untrusted context, validate outputs, and test.

Mistake: “The provider never trains on our inputs, so there is no data risk.”
Why: Training is only one part of retention, access, logging, and contract review.
Fix: Review the actual service terms and configuration with the data owner.

Mistake: “The model returned valid JSON, so the action is approved.”
Why: Syntactic validity does not prove identity, eligibility, or correctness.
Fix: Apply business rules and authorization in trusted application code.

Practice: Review a Safe AI Feature

1. Draw a data-flow diagram for a course-support chatbot and mark user, retrieval, model, tools, and logs as trust boundaries. 2. Add a synthetic retrieved document containing irrelevant instructions to a local test fixture; confirm your application still limits retrieval to the test user's records. 3. List three sensitive fields that are unnecessary for a course recommendation and remove them from a mock model request. 4. Design a schema-validation and human-confirmation step for a model-proposed account change. 5. Write an abuse-monitoring signal that records request counts and cost without storing full sensitive prompts. 6. Record one limitation of each control and an owner responsible for reviewing it.

🎯 Key Takeaways

  • ✓Prompt injection is a trust-boundary problem; prompt wording alone cannot enforce permissions.
  • ✓Authorise retrieval and tools in trusted application code using the current user and tenant context.
  • ✓Minimise sensitive data sent to models and review provider retention, access, and logging settings.
  • ✓Treat generated text and structured output as untrusted; validate, encode, and apply business rules.
  • ✓Use least privilege, rate limits, auditability, and human confirmation for consequential actions.
  • ✓Test with synthetic data in an isolated environment and reassess risks when capabilities change.
💡 Note
Next in Cybersecurity — Lesson 39: Mobile Application Security
Continue with mobile threat modelling, OWASP MASVS, and practical Android and iOS review patterns. Continue to Lesson 39 →
🎯 Pro Tip
Want the complete AI Security learning path?
This lesson is only an introduction. Continue to the full course for deeper coverage of AI threat modelling, secure design, evaluation, privacy, and operations. Explore the full AI Security course →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...