Observability and Tracing for LLM Apps

Trace every model call, retrieval, and tool call with OpenTelemetry-style spans; track cost, latency, and quality in production.

Est. 30 min Updated Suggest a fix
// Part 01 — Why observability

Answer the Question: What Happened to This Request?

When an AI response is slow, incorrect, or unexpectedly expensive, the browser may show only one request and one error. The actual work can include authentication, retrieval, reranking, a model call, and one or more tool calls. Observability connects those steps so an engineer can follow a request from its entry point to its final response.

A trace is the record of one end-to-end operation. It contains spans: timed units of work with a parent-child relationship. A span might represent a request, a vector search, an LLM call, or a tool execution. Attributes such as model configuration, status, token counts, and request identifiers help you investigate without needing to log the full prompt or answer.

SignalWhat it tells youLLM example
TraceHow operations relate across a requestAPI request → retrieval → model → response
SpanDuration and outcome of one operationSearch took longer than the generation call
MetricAggregated behavior over timep95 latency or requests with provider errors
Log / eventA discrete fact tied to a requestProvider timeout with a correlation ID
💡 Note
OpenTelemetry provides vendor-neutral concepts and interfaces for traces, metrics, and logs. A tracing backend stores, searches, and visualizes the data. Instrument once around meaningful operations, then choose a compatible backend.
// Part 02 — Trace structure

Give Every Request a Trace and Useful Spans

Start a trace at the application boundary and propagate its context through internal calls. Give spans names that describe the operation rather than user content: retrieval.search or llm.generate. Record start time, end time, status, and a small set of bounded attributes. The parent-child structure exposes where total time was spent and which dependency failed.

  1. Create or propagate a request correlation ID at the API boundary.
  2. Create child spans for retrieval, reranking, model calls, and tools.
  3. Attach safe attributes such as deployment environment, operation, and result status.
  4. Record durations and token usage when returned by the provider.
  5. Set an error status and preserve a safe error category when a step fails.
trace_core.py
python
from contextlib import contextmanager
from dataclasses import dataclass, field
from time import perf_counter
from typing import Iterator

@dataclass
class Span:
    name: str
    attributes: dict[str, str | int | float] = field(default_factory=dict)
    started_at: float = field(default_factory=perf_counter)
    error_type: str | None = None

    def set_attribute(self, key: str, value: str | int | float) -> None:
        self.attributes[key] = value

@contextmanager
def span(name: str) -> Iterator[Span]:
    current = Span(name=name)
    try:
        yield current
    except Exception as error:
        current.error_type = type(error).__name__
        raise
    finally:
        elapsed_ms = (perf_counter() - current.started_at) * 1000
        print({"span": current.name, "duration_ms": round(elapsed_ms, 1),
               "error_type": current.error_type, "attributes": current.attributes})

with span("rag.answer") as request_span:
    with span("retrieval.search") as retrieval_span:
        retrieval_span.set_attribute("retrieval.top_k", 4)
    with span("llm.generate") as model_span:
        model_span.set_attribute("llm.provider", "configured-provider")

This standard-library example prints one span at a time; it is an observability concept, not a production tracing exporter. In an instrumented service, use OpenTelemetry APIs and an exporter supported by your chosen collector or backend so context and parent relationships are propagated automatically.

// Part 03 — Model-call spans

Record Usage and Latency Without Recording Everything

Model-call spans are especially useful because generation time, token use, provider outcome, and configured model affect both experience and cost. Read the model name from configuration; do not bake a particular model version into instrumentation. Capture provider request IDs where available so an operations team can correlate an incident with a provider support request.

instrumented_call.py
python
import os
from time import perf_counter
from openai import OpenAI

client = OpenAI(timeout=30.0)

def answer(question: str) -> tuple[str, dict[str, int | float | str]]:
    started = perf_counter()
    model = os.environ["LLM_MODEL"]
    try:
        response = client.responses.create(
            model=model,
            instructions="Answer briefly using the supplied application context.",
            input=question,
            max_output_tokens=300,
        )
        usage = response.usage
        attributes = {
            "llm.model_config": model,
            "llm.input_tokens": usage.input_tokens,
            "llm.output_tokens": usage.output_tokens,
            "llm.duration_ms": round((perf_counter() - started) * 1000, 1),
        }
        return response.output_text, attributes
    except Exception as error:
        # Production code should map known provider failures to safe categories.
        raise RuntimeError("Model operation failed") from error

Integrate those attributes with the active span rather than printing them in a real service. The example uses a familiar SDK response shape, but exact fields may vary by SDK version; follow the installed SDK’s current official documentation. Avoid catching every exception and converting it into success: record a safe failure status and let the API return an intentional error response.

⚠️ Important
Prompts, completions, retrieved passages, and user identifiers can contain sensitive data. Disable content capture by default or apply a reviewed redaction and access policy before storing any content in a tracing product.
// Part 04 — Metrics

Use Metrics to See Patterns, Not Just Individual Requests

A trace is excellent for debugging one request. Metrics help answer whether a service is getting worse across many requests. Track request volume, errors by safe category, provider timeouts, tokens, and end-to-end latency. Percentiles such as p50 and p95 show typical and tail experience better than a single mean that can hide slow requests.

MetricUseful viewQuestion
Request durationp50, p95, and trend by routeAre users waiting longer?
Model failuresRate by provider and safe error classIs a dependency unavailable or throttling?
Token usageInput/output distribution by featureDid a prompt or history grow unexpectedly?
Quality signalEval score, feedback, or escalation rateDid a release change answer usefulness?
Trace samplingRepresentative traces by route/statusCan an engineer inspect the slow path?

Use consistent units and bounded dimensions. Route, deployment, provider, and outcome categories are usually more useful than raw user IDs, full prompts, or arbitrary query text. Unbounded metric labels can overwhelm a metrics system and create a privacy risk.

// Part 05 — OpenTelemetry

Understand the Concepts Before Choosing a Backend

OpenTelemetry instrumentation creates telemetry signals and exports them to a collector or backend. A collector can receive, process, and route data; a backend stores and presents traces, metrics, and logs. This separation helps avoid coupling application code to one observability vendor. Configure sampling, redaction, and export credentials as part of the service deployment.

LLM-focused products such as Langfuse and LangSmith provide workflows for inspecting model calls, prompt versions, evaluations, and traces. Their capabilities and interfaces change, so compare current documentation before selecting one. Treat any vendor as a data processor: review what is captured, where it is stored, who can access it, and how long it is retained.

🎯 Pro Tip
Keep the application’s tracing boundary small and vendor-neutral. Add backend-specific dashboards and evaluations without scattering vendor-specific calls across every business function.
// Part 06 — Redaction and sampling

Make Telemetry Useful Without Making It a Data Leak

Observability needs enough detail to diagnose behavior, but raw content may include personal, financial, or confidential information. Prefer a request ID, operation name, token counts, latency, status, and a small category over complete input/output capture. If content is necessary for debugging, use a limited, approved sampling path with redaction, access restrictions, and a retention schedule.

safe_event.py
python
from dataclasses import dataclass
from time import perf_counter

@dataclass
class CallSummary:
    request_id: str
    operation: str
    model_setting: str
    input_tokens: int
    output_tokens: int
    duration_ms: float
    outcome: str

def summarize(request_id: str, operation: str, model_setting: str,
              input_tokens: int, output_tokens: int,
              started: float, outcome: str) -> CallSummary:
    # Deliberately omit prompt, completion, email, phone, and account data.
    return CallSummary(
        request_id=request_id,
        operation=operation,
        model_setting=model_setting,
        input_tokens=input_tokens,
        output_tokens=output_tokens,
        duration_ms=round((perf_counter() - started) * 1000, 1),
        outcome=outcome,
    )

started = perf_counter()
summary = summarize("req-123", "support-answer", "configured-model", 90, 42, started, "ok")
print(summary)

Sampling can control volume, but do not sample away all errors. Consider retaining error traces more often and sampling healthy traces at a lower rate, subject to your privacy policy. Verify that debug modes cannot be enabled by an untrusted browser parameter or left on after an incident.

// Part 07 — Operational workflow

Turn Traces into a Debugging Habit

When someone reports a bad answer, begin with a correlation ID and trace the request path. Check which context was retrieved, whether tool calls succeeded, how long the model call took, and whether the response passed validation. Compare the trace with an evaluation case or known-good request. Keep incident notes focused on the cause and the preventive test, not copied personal content.

  1. Locate the trace by request ID or a privacy-safe time and route filter.
  2. Find the slowest or failed span and inspect its status and safe attributes.
  3. Separate retrieval, provider, application, and rendering failures.
  4. Check token and latency trends against a known baseline and recent deployment.
  5. Add a regression test or alert for the failure mode, then close the incident with data-retention rules intact.

Alert on symptoms a team can act on, such as sustained error rate or tail latency, not every single unusual output. Pair operational monitoring with quality evaluation and user feedback: a fast, successful HTTP request can still contain a wrong answer.

📌 Real World Example
If an assistant’s p95 response time rises after a release, a trace can show whether retrieval, provider generation, or an added validation step accounts for the change. The fix depends on the span, not on guessing from the final page.
// Part 08 — Pitfalls

Common Observability Mistakes

Logging every prompt and completion by default

Why: Content may contain personal or confidential data.

Fix: Capture minimal metadata; make content capture a reviewed, restricted exception.

Recording a single end-to-end duration only

Why: The trace cannot show which dependency caused the delay.

Fix: Create spans around retrieval, model calls, tools, and validation.

Using user text as metric labels

Why: Labels become unbounded and can expose sensitive information.

Fix: Use low-cardinality categories and keep content out of metric dimensions.

Assuming an HTTP 200 means a useful answer

Why: Transport success does not measure factuality or policy compliance.

Fix: Connect traces to evaluations, feedback, and human escalation signals.

// Part 09 — Practice

Instrument a Small Retrieval Assistant

Practice

Trace the path while protecting user content

  1. Add a request correlation ID and spans for retrieval, generation, and output validation in a local prototype.
  2. Record duration, token counts, operation, configured model setting, and safe outcome status for each model call.
  3. Write down which fields are sensitive and ensure prompt and completion text are absent from ordinary logs.
  4. Create a dashboard specification for request volume, errors, p50/p95 latency, and token usage by feature.
  5. Simulate a provider timeout and use the trace to identify the failed span and safe user-facing fallback.

Hint: Start with the trace tree and attributes you need to debug one request. Exporter and dashboard choices can follow.

🎯 Key Takeaways

  • ✓A trace connects a request; spans represent timed child operations such as retrieval, model generation, and tools.
  • ✓OpenTelemetry offers vendor-neutral concepts for traces, metrics, and logs, while a backend stores and displays them.
  • ✓Capture latency, token usage, outcome, and safe correlation metadata to investigate production behavior.
  • ✓Use p50 and p95 latency, error rates, and quality signals to see patterns across requests.
  • ✓Treat prompts and completions as potentially sensitive; minimise, redact, restrict, and retain telemetry deliberately.
  • ✓A successful request is not necessarily a good answer—combine operational traces with evaluation and user feedback.
  • Consult OpenTelemetry documentation and the current Langfuse or LangSmith documentation for their supported integrations, data handling, and configuration options.

Next

What comes next

You can now see where time and tokens go. Next, use that evidence to reduce cost and improve response speed without sacrificing quality.

Lesson 11 · Cost and Latency Optimisation — Caching, Batching and Routing →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...