Sensitive Data, Privacy and Output Handling

Keep PII and secrets out of prompts, logs and training data, treat model output as untrusted input, and meet privacy obligations such as the DPDP Act.

Est. 35 min Updated Suggest a fix
// Part 01 — Privacy boundaries

An AI Feature Creates More Copies of Data

A request to an AI feature may pass through a browser, your application server, a retrieval service, an observability pipeline and a model provider. A customer support assistant for a UPI payments app might handle a name, an order reference and a conversation that unexpectedly includes a phone number or account detail. Each copy and each recipient is a place where data can be exposed, retained longer than intended or made available to someone who should not see it.

Start by tracing the data, not by choosing a masking library. Identify which fields are necessary for the task, where they enter, where they leave your control, who can access them, how long they persist and what happens when the user asks for a deletion or correction. Separate personal data from credentials and other secrets: both need protection, but an API key should never be sent to a model as if it were ordinary user content.

Data pathQuestion to askSafer default
User inputCould the field contain unexpected personal or payment data?Collect only what the task needs; show a clear input boundary
Prompt and retrievalDoes context include another person or tenant?Filter by authorization before retrieval and minimize context
Provider requestWhat is sent to an external service and under what terms?Review provider configuration and approved data classes
Logs and tracesCan operators read raw prompts or outputs?Record metadata; redact or avoid sensitive content
Output and toolsCan generated text trigger a consequential action?Treat it as untrusted; validate and authorize downstream use
💡 Note
A privacy review is a data-flow exercise. A promise that a prompt is "private" does not replace checking the actual product configuration, provider terms, retention controls, subprocessors and access permissions.
// Part 02 — Minimize first

Send the Smallest Useful Context

Data minimization is often stronger than trying to redact everything after it has already been copied. If a delivery assistant only needs to explain a delayed order, provide an internal order status and a coarse time window—not the customer’s full profile, address, payment credentials or unrelated chat history. Where identity is required, use a short-lived internal reference and resolve it inside a trusted service rather than placing identifying data in a prompt.

  1. Map fields from the user, database, retrieved documents, tools and model response.
  2. For each field, state the purpose and the minimum precision needed; remove fields with no clear purpose.
  3. Decide whether the model actually needs the original value or whether a category, range or opaque reference works.
  4. Set access, retention and deletion expectations for every system that receives the data.
🎯 Pro Tip
Tokenization is not automatically anonymization. A stable identifier can still be linkable to a person if another system can map it back. Protect the mapping and limit where the token can be used.
// Part 03 — Privacy-aware code

Minimize and Redact Before the Provider Boundary

The small Python example below demonstrates a conservative allow-list for fields and a narrow redaction step. It is deliberately not presented as a complete PII detector: patterns miss names, context-dependent identifiers and new formats. For sensitive workflows, prefer structured inputs, dedicated detection services where appropriate, human review for uncertain matches and an explicit failure path.

privacy_boundary.py
python
import re
from typing import Any

# This demo recognizes only a few illustrative formats; it is not a full PII detector.
PHONE = re.compile(r"(?<!\d)(?:\+91[ -]?)?[6-9]\d{9}(?!\d)")
EMAIL = re.compile(r"(?i)\b[\w.+-]+@[\w.-]+\.[a-z]{2,}\b")


def minimize_order_context(order: dict[str, Any]) -> dict[str, str]:
    allowed = {"status", "estimated_window"}
    return {
        key: str(value)
        for key, value in order.items()
        if key in allowed and value is not None
    }


def redact_obvious_contact_details(text: str) -> str:
    text = EMAIL.sub("[EMAIL]", text)
    return PHONE.sub("[PHONE]", text)


customer_text = "My order is late; call me at +91 9876543210"
order = {"status": "in transit", "estimated_window": "this evening", "address": "ignored"}
model_input = {
    "question": redact_obvious_contact_details(customer_text),
    "order": minimize_order_context(order),
}
print(model_input)

Notice two separate controls: the allow-list avoids sending an unnecessary address, while redaction reduces exposure in free-form text. Neither control grants authorization. The server must still verify that the signed-in customer is entitled to access that order before it constructs the context.

// Part 04 — Logging and retention

Observe the System Without Keeping Every Prompt

Teams need enough telemetry to find failures and investigate misuse, but a full prompt-and-response archive is not the only way to do that. Keep operational metadata such as a request identifier, model configuration reference, latency, token counts, policy decision and outcome. If sampled content is necessary for quality review, document its purpose, restrict access, redact carefully, define retention and make the collection transparent to the people responsible for it.

safe_event.py
python
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
import json

@dataclass
class ModelEvent:
    request_id: str
    created_at: str
    model_config: str
    input_tokens: int
    output_tokens: int
    policy_result: str
    outcome: str


def log_event(event: ModelEvent) -> None:
    # Deliberately omit raw prompt, completion, user ID, and provider credentials.
    print(json.dumps(asdict(event), separators=(",", ":")))


event = ModelEvent(
    request_id="req-example-001",
    created_at=datetime.now(timezone.utc).isoformat(),
    model_config="support-assistant-prod",
    input_tokens=180,
    output_tokens=42,
    policy_result="allowed",
    outcome="answered",
)
log_event(event)

Use a random request ID for correlation and keep any identity mapping in a separately protected system. Check error handlers, traces, analytics events, support dashboards and crash reports too: data often leaks through diagnostics that were not designed as a content store. A retention period should be justified and enforced, not merely written down.

// Part 05 — Output is untrusted

Validate Before Rendering, Saving or Acting

A model can return malformed JSON, unsafe HTML, a false claim or text that looks like an instruction. Its output has no special trust merely because it came from an approved model. Encode text for the rendering context, parse structured values with a strict schema, enforce bounds and check authorization in ordinary application code. Never treat a model statement such as "the user is an administrator" as proof of identity or permission.

validate_output.py
python
import json
from pydantic import BaseModel, ConfigDict, Field, ValidationError

class SupportReply(BaseModel):
    model_config = ConfigDict(extra="forbid")
    message: str = Field(min_length=1, max_length=1200)
    order_reference: str | None = Field(default=None, max_length=40)


def parse_reply(raw: str) -> SupportReply:
    # Parsing validates shape and bounds; it does not make claims true.
    return SupportReply.model_validate_json(raw)


def reply_for_customer(raw: str, customer_id: str) -> str:
    try:
        reply = parse_reply(raw)
    except ValidationError as error:
        raise ValueError("The assistant returned an invalid response") from error

    if reply.order_reference is not None:
        # A real application must fetch the order and authorize ownership here.
        if not customer_id:
            raise PermissionError("Authentication required")
    return reply.message


candidate = '{"message":"Your order is being checked.","order_reference":null}'
print(reply_for_customer(candidate, customer_id="customer-123"))

If output is inserted into HTML, use the framework’s escaped text rendering instead of concatenating markup. If it is used in SQL, use parameterized queries. If it could initiate a refund or change an account, require a trusted service to authorize the action and apply transaction limits independently of the model.

⚠️ Important
Schema-valid does not mean safe, private or correct. Validate semantics, permissions and business rules after parsing; retain a safe fallback when validation fails.
// Part 06 — Privacy obligations

Make Compliance a Designed Workflow

In India, the Digital Personal Data Protection Act, 2023 is a relevant part of the privacy landscape for digital personal data. Whether and how particular duties apply depends on the facts, applicable rules and current official guidance. Do not treat this lesson as legal advice or assume that a model provider’s setting alone settles your organization’s responsibilities. Ask the appropriate privacy or legal owner to review purpose, notice, access, retention, transfers, vendor roles and processes for requests or incidents.

Translate the review into engineering controls: documented data inventory, purpose-linked fields, least-privilege access, approved vendors, deletion paths, data-subject request handling and incident escalation. For user-facing AI, make it clear what the feature does and avoid promising that generated content is a verified decision. Keep evidence of the design and review so changes in prompts, providers or logging can be reassessed.

  • Assign an owner for each data flow and provider integration.
  • Record what is sent, why it is needed, and which environment or vendor receives it.
  • Test deletion and access-control paths across caches, logs, indexes and backups according to approved policy.
  • Escalate a suspected exposure promptly through the organization’s incident process; do not silently erase evidence.
// Part 07 — Pitfalls

Common Mistakes

Assuming a regex removes all personal data

Why: Identifiers vary by format and context, and users can disclose unexpected details.

Fix: Minimize structured fields first; use layered detection and test misses with representative synthetic cases.

Logging complete prompts for convenience

Why: Logs and traces often have broader access and longer retention than the primary database.

Fix: Log operational metadata by default; limit, protect and expire content samples when a justified need exists.

Calling a pseudonymous ID anonymous

Why: A separate mapping or repeated usage may make a person identifiable again.

Fix: Treat linkable tokens as sensitive, separate the mapping and limit purpose and access.

Trusting schema-valid model output

Why: A schema checks shape, not truth, authorization or safe downstream behavior.

Fix: Validate business rules and ownership in trusted code; encode for the output context.

// Part 08 — Practice and review

Trace One Sensitive Data Flow

Practice

Audit a fictional order-status assistant

  1. Draw the path from browser input through retrieval, provider, logs and response rendering; mark trust boundaries.
  2. Choose the minimum order fields needed to answer a delivery-status question and justify each one.
  3. Create synthetic examples with phone numbers, emails and an unexpected credential; note what your detector misses.
  4. Design a metadata-only event record and a retention/access policy for any approved review sample.
  5. Define output schema checks plus the server-side ownership check before showing an order-specific response.

Hint: Use fake values only. A successful test must also show that another customer’s record is never retrieved or logged.

🎯 Key Takeaways

  • ✓Map personal data, secrets, recipients, access and retention before adding a model call.
  • ✓Minimize the context at its source; redaction is a secondary control, not a substitute.
  • ✓Keep prompts and completions out of routine logs; correlate with protected request identifiers.
  • ✓Treat generated text and structured output as untrusted until parsed and checked against application rules.
  • ✓Pseudonymized values can remain personal when they are linkable to a person.
  • ✓Review privacy obligations with current official guidance and qualified owners; build controls into the product.

Next

What comes next

Next, examine how third-party models, datasets and packages enter an AI system—and how to verify provenance and limit poisoning risk.

Lesson 07 · Model Supply Chain and Data Poisoning →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...