Observability and Tracing for LLM Apps
Trace every model call, retrieval, and tool call with OpenTelemetry-style spans; track cost, latency, and quality in production.
Answer the Question: What Happened to This Request?
When an AI response is slow, incorrect, or unexpectedly expensive, the browser may show only one request and one error. The actual work can include authentication, retrieval, reranking, a model call, and one or more tool calls. Observability connects those steps so an engineer can follow a request from its entry point to its final response.
A trace is the record of one end-to-end operation. It contains spans: timed units of work with a parent-child relationship. A span might represent a request, a vector search, an LLM call, or a tool execution. Attributes such as model configuration, status, token counts, and request identifiers help you investigate without needing to log the full prompt or answer.
| Signal | What it tells you | LLM example |
|---|---|---|
| Trace | How operations relate across a request | API request → retrieval → model → response |
| Span | Duration and outcome of one operation | Search took longer than the generation call |
| Metric | Aggregated behavior over time | p95 latency or requests with provider errors |
| Log / event | A discrete fact tied to a request | Provider timeout with a correlation ID |
Give Every Request a Trace and Useful Spans
Start a trace at the application boundary and propagate its context through internal calls. Give spans names that describe the operation rather than user content: retrieval.search or llm.generate. Record start time, end time, status, and a small set of bounded attributes. The parent-child structure exposes where total time was spent and which dependency failed.
- Create or propagate a request correlation ID at the API boundary.
- Create child spans for retrieval, reranking, model calls, and tools.
- Attach safe attributes such as deployment environment, operation, and result status.
- Record durations and token usage when returned by the provider.
- Set an error status and preserve a safe error category when a step fails.
from contextlib import contextmanager
from dataclasses import dataclass, field
from time import perf_counter
from typing import Iterator
@dataclass
class Span:
name: str
attributes: dict[str, str | int | float] = field(default_factory=dict)
started_at: float = field(default_factory=perf_counter)
error_type: str | None = None
def set_attribute(self, key: str, value: str | int | float) -> None:
self.attributes[key] = value
@contextmanager
def span(name: str) -> Iterator[Span]:
current = Span(name=name)
try:
yield current
except Exception as error:
current.error_type = type(error).__name__
raise
finally:
elapsed_ms = (perf_counter() - current.started_at) * 1000
print({"span": current.name, "duration_ms": round(elapsed_ms, 1),
"error_type": current.error_type, "attributes": current.attributes})
with span("rag.answer") as request_span:
with span("retrieval.search") as retrieval_span:
retrieval_span.set_attribute("retrieval.top_k", 4)
with span("llm.generate") as model_span:
model_span.set_attribute("llm.provider", "configured-provider")This standard-library example prints one span at a time; it is an observability concept, not a production tracing exporter. In an instrumented service, use OpenTelemetry APIs and an exporter supported by your chosen collector or backend so context and parent relationships are propagated automatically.
Record Usage and Latency Without Recording Everything
Model-call spans are especially useful because generation time, token use, provider outcome, and configured model affect both experience and cost. Read the model name from configuration; do not bake a particular model version into instrumentation. Capture provider request IDs where available so an operations team can correlate an incident with a provider support request.
import os
from time import perf_counter
from openai import OpenAI
client = OpenAI(timeout=30.0)
def answer(question: str) -> tuple[str, dict[str, int | float | str]]:
started = perf_counter()
model = os.environ["LLM_MODEL"]
try:
response = client.responses.create(
model=model,
instructions="Answer briefly using the supplied application context.",
input=question,
max_output_tokens=300,
)
usage = response.usage
attributes = {
"llm.model_config": model,
"llm.input_tokens": usage.input_tokens,
"llm.output_tokens": usage.output_tokens,
"llm.duration_ms": round((perf_counter() - started) * 1000, 1),
}
return response.output_text, attributes
except Exception as error:
# Production code should map known provider failures to safe categories.
raise RuntimeError("Model operation failed") from errorIntegrate those attributes with the active span rather than printing them in a real service. The example uses a familiar SDK response shape, but exact fields may vary by SDK version; follow the installed SDK’s current official documentation. Avoid catching every exception and converting it into success: record a safe failure status and let the API return an intentional error response.
Use Metrics to See Patterns, Not Just Individual Requests
A trace is excellent for debugging one request. Metrics help answer whether a service is getting worse across many requests. Track request volume, errors by safe category, provider timeouts, tokens, and end-to-end latency. Percentiles such as p50 and p95 show typical and tail experience better than a single mean that can hide slow requests.
| Metric | Useful view | Question |
|---|---|---|
| Request duration | p50, p95, and trend by route | Are users waiting longer? |
| Model failures | Rate by provider and safe error class | Is a dependency unavailable or throttling? |
| Token usage | Input/output distribution by feature | Did a prompt or history grow unexpectedly? |
| Quality signal | Eval score, feedback, or escalation rate | Did a release change answer usefulness? |
| Trace sampling | Representative traces by route/status | Can an engineer inspect the slow path? |
Use consistent units and bounded dimensions. Route, deployment, provider, and outcome categories are usually more useful than raw user IDs, full prompts, or arbitrary query text. Unbounded metric labels can overwhelm a metrics system and create a privacy risk.
Understand the Concepts Before Choosing a Backend
OpenTelemetry instrumentation creates telemetry signals and exports them to a collector or backend. A collector can receive, process, and route data; a backend stores and presents traces, metrics, and logs. This separation helps avoid coupling application code to one observability vendor. Configure sampling, redaction, and export credentials as part of the service deployment.
LLM-focused products such as Langfuse and LangSmith provide workflows for inspecting model calls, prompt versions, evaluations, and traces. Their capabilities and interfaces change, so compare current documentation before selecting one. Treat any vendor as a data processor: review what is captured, where it is stored, who can access it, and how long it is retained.
Make Telemetry Useful Without Making It a Data Leak
Observability needs enough detail to diagnose behavior, but raw content may include personal, financial, or confidential information. Prefer a request ID, operation name, token counts, latency, status, and a small category over complete input/output capture. If content is necessary for debugging, use a limited, approved sampling path with redaction, access restrictions, and a retention schedule.
from dataclasses import dataclass
from time import perf_counter
@dataclass
class CallSummary:
request_id: str
operation: str
model_setting: str
input_tokens: int
output_tokens: int
duration_ms: float
outcome: str
def summarize(request_id: str, operation: str, model_setting: str,
input_tokens: int, output_tokens: int,
started: float, outcome: str) -> CallSummary:
# Deliberately omit prompt, completion, email, phone, and account data.
return CallSummary(
request_id=request_id,
operation=operation,
model_setting=model_setting,
input_tokens=input_tokens,
output_tokens=output_tokens,
duration_ms=round((perf_counter() - started) * 1000, 1),
outcome=outcome,
)
started = perf_counter()
summary = summarize("req-123", "support-answer", "configured-model", 90, 42, started, "ok")
print(summary)Sampling can control volume, but do not sample away all errors. Consider retaining error traces more often and sampling healthy traces at a lower rate, subject to your privacy policy. Verify that debug modes cannot be enabled by an untrusted browser parameter or left on after an incident.
Turn Traces into a Debugging Habit
When someone reports a bad answer, begin with a correlation ID and trace the request path. Check which context was retrieved, whether tool calls succeeded, how long the model call took, and whether the response passed validation. Compare the trace with an evaluation case or known-good request. Keep incident notes focused on the cause and the preventive test, not copied personal content.
- Locate the trace by request ID or a privacy-safe time and route filter.
- Find the slowest or failed span and inspect its status and safe attributes.
- Separate retrieval, provider, application, and rendering failures.
- Check token and latency trends against a known baseline and recent deployment.
- Add a regression test or alert for the failure mode, then close the incident with data-retention rules intact.
Alert on symptoms a team can act on, such as sustained error rate or tail latency, not every single unusual output. Pair operational monitoring with quality evaluation and user feedback: a fast, successful HTTP request can still contain a wrong answer.
Common Observability Mistakes
Logging every prompt and completion by default
Why: Content may contain personal or confidential data.
Fix: Capture minimal metadata; make content capture a reviewed, restricted exception.
Recording a single end-to-end duration only
Why: The trace cannot show which dependency caused the delay.
Fix: Create spans around retrieval, model calls, tools, and validation.
Using user text as metric labels
Why: Labels become unbounded and can expose sensitive information.
Fix: Use low-cardinality categories and keep content out of metric dimensions.
Assuming an HTTP 200 means a useful answer
Why: Transport success does not measure factuality or policy compliance.
Fix: Connect traces to evaluations, feedback, and human escalation signals.
Instrument a Small Retrieval Assistant
Practice
Trace the path while protecting user content
- Add a request correlation ID and spans for retrieval, generation, and output validation in a local prototype.
- Record duration, token counts, operation, configured model setting, and safe outcome status for each model call.
- Write down which fields are sensitive and ensure prompt and completion text are absent from ordinary logs.
- Create a dashboard specification for request volume, errors, p50/p95 latency, and token usage by feature.
- Simulate a provider timeout and use the trace to identify the failed span and safe user-facing fallback.
Hint: Start with the trace tree and attributes you need to debug one request. Exporter and dashboard choices can follow.
🎯 Key Takeaways
- ✓A trace connects a request; spans represent timed child operations such as retrieval, model generation, and tools.
- ✓OpenTelemetry offers vendor-neutral concepts for traces, metrics, and logs, while a backend stores and displays them.
- ✓Capture latency, token usage, outcome, and safe correlation metadata to investigate production behavior.
- ✓Use p50 and p95 latency, error rates, and quality signals to see patterns across requests.
- ✓Treat prompts and completions as potentially sensitive; minimise, redact, restrict, and retain telemetry deliberately.
- ✓A successful request is not necessarily a good answer—combine operational traces with evaluation and user feedback.
- Consult OpenTelemetry documentation and the current Langfuse or LangSmith documentation for their supported integrations, data handling, and configuration options.
Next
What comes next
You can now see where time and tokens go. Next, use that evidence to reduce cost and improve response speed without sacrificing quality.
Lesson 11 · Cost and Latency Optimisation — Caching, Batching and Routing →HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
