Monitoring, Logging and Alerting
Understand metrics, logs, and traces; create useful Prometheus and Grafana views; set service-level objectives; and respond to alerts with a sustainable on-call practice.
Answer the User Question Before the Infrastructure Question
At 9:15 AM, learners report that a course dashboard is taking too long to open. The first useful question is not "is the CPU high?" It is "are users affected, and where is the request getting slow?" Monitoring turns system behavior into evidence an operator can use to detect impact, diagnose causes, and decide whether to mitigate or escalate.
Three signals complement one another: metrics summarize numeric behavior over time, logs record discrete events with context, and traces connect the work performed across services for one request. A metric can tell you that latency has increased; a trace can show which dependency added time; logs can provide the relevant error details. None of the three is sufficient on its own for every investigation.
| Signal | Best for | Example |
|---|---|---|
| Metrics | Trends, rates, saturation, and alert conditions | Request rate, error ratio, queue depth, CPU use |
| Logs | Detailed events and error context | A failed order lookup with request ID and error category |
| Traces | Following one request across service boundaries | Browser API → order service → database call |
Measure Rates, Errors, Latency and Saturation
A useful service dashboard often begins with traffic, errors, latency, and saturation. Traffic describes demand; errors describe failed work; latency describes how long work takes; saturation shows whether a constrained resource is nearing capacity. Track these at the service boundary and, where useful, at key dependencies such as a database or queue. Use units, names, and label conventions consistently.
Counters increase over time, gauges can rise or fall, and histograms record observations in buckets that support latency distributions and percentiles. Averages can hide a slow tail: a small number of very slow requests may make the user experience poor even when the average looks acceptable. Choose histogram buckets and aggregation carefully, and compare like-for-like request classes.
| Metric idea | Question it helps answer | Caution |
|---|---|---|
| Request rate | How much work is arriving? | Break down only by useful, bounded dimensions. |
| Error ratio | What share of requests fail? | Distinguish expected client errors from service failures. |
| Latency distribution | How slow are typical and tail requests? | Averages alone hide outliers and slow user journeys. |
| Saturation | Which resource is near its useful limit? | CPU alone may miss memory, connection, queue, or downstream limits. |
Scrape Metrics and Evaluate Rules
Prometheus commonly collects metrics by scraping an HTTP endpoint exposed by an application or exporter. The application must instrument the measurements; Prometheus cannot infer meaningful business events from a container alone. A scrape configuration names a job and targets that expose a metrics endpoint. Service discovery and authentication vary by environment; this small example is a starting point for a local lab, not a production security configuration.
global:
scrape_interval: 30s
rule_files:
- "alerts.yml"
scrape_configs:
- job_name: "orders-api"
metrics_path: "/metrics"
static_configs:
- targets: ["orders-api:80"]Prometheus stores labeled time series. A metric such as http_requests_total is a counter emitted by the application; labels might include a bounded route name and status class. PromQL can calculate a per-second rate over a time window and aggregate it across instances. Confirm metric names and labels in the actual service before writing a dashboard or rule—custom instrumentation determines what is available.
From a local lab where the Prometheus HTTP API is reachable on the loopback interface, query the standard up metric for this scrape job. A successful response means Prometheus has a recent sample for the target; it does not prove that the application is healthy for users.
curl --fail --get 'http://localhost:9090/api/v1/query' --data-urlencode 'query=up{job="orders-api"}'groups:
- name: orders-api
rules:
- alert: ElevatedServerErrorRatio
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
clamp_min(sum(rate(http_requests_total[5m])), 1)
> 0.02
for: 10m
labels:
severity: warning
annotations:
summary: "Orders API server error ratio is elevated"
description: "Check the service dashboard and recent deploys."The two-percent expression is only a teaching threshold; it is not a universal SLO. A real alert should reflect customer impact, baseline traffic, and the service's agreed objectives. The rule also assumes the counter exists and that the status label has the expected values. Test the expression against real series before routing pages.
Keep Context Useful, Safe and Connected
Structured logs make fields searchable: timestamp, severity, service, environment, event name, and a request or trace identifier. Log the reason a request failed without dumping credentials, authorization headers, full payment data, or unnecessary personal information. Define retention and access based on operational need and applicable policy; logs are data stores and deserve security controls.
Distributed tracing represents a request as spans across services. Instrumenting a service with OpenTelemetry or a compatible library lets teams propagate trace context and export spans to a collector or backend. A trace ID in logs helps move from a metric alert to the exact request path. Sampling can control volume, but ensure the sampling policy does not systematically hide the rare failures you need to investigate.
| Question | Signal to start with | Next step |
|---|---|---|
| Did user-facing errors rise? | Error-rate metric | Filter by route/status and inspect a representative trace. |
| Which request failed and why? | Structured log | Follow the trace ID and inspect related service spans. |
| Which dependency added latency? | Trace waterfall | Compare span timing and dependency metrics. |
| Is the issue widespread or one request? | Metrics by bounded service/route dimension | Use logs and traces for per-request detail. |
Build Dashboards Around Decisions
Grafana can query Prometheus and other data sources, render panels, and help operators move from a high-level service view to a focused investigation. A useful dashboard answers a few operational questions at a glance: is demand normal, are users seeing errors, is latency within target, and which dependency is saturated? Add links to logs, traces, and recent deployments so the next diagnostic step is clear.
Make panels consistent in units and time ranges, add descriptions, and use labels that separate environment and service without exposing sensitive dimensions. Avoid a wall of gauges that nobody uses. Review dashboards after incidents and remove panels that are redundant or misleading. Keep access controls aligned with the sensitivity of the underlying data.
- Start with service traffic, errors, latency distribution, and saturation.
- Add the most important dependencies and queue/backlog signals.
- Show deployment markers or a link to release history when available.
- Link a panel to the relevant logs and traces for deeper investigation.
- Ask an on-call teammate to use the dashboard during a drill and improve unclear panels.
Define Reliability from the User Perspective
A Service Level Indicator (SLI) is a measured aspect of service behavior, such as the proportion of valid requests completed successfully or a latency distribution. A Service Level Objective (SLO) is the target for an SLI over a defined time window. A Service Level Agreement (SLA) is an external commitment that may have business consequences; an internal SLO is not automatically an SLA. Choose indicators users care about rather than optimizing an arbitrary host metric.
For a course platform, possible SLIs include successful lesson-page loads, completion of a practice submission, and latency for those actions. Define which requests count, how synthetic checks are treated, and how the measurement is aggregated. A target such as 99.9% is only an example until the product team chooses a realistic objective based on user needs, architecture, and evidence. An error budget is the remaining allowed unreliability implied by the SLO; teams can use it to balance release speed and reliability work.
| Term | Definition | Example question |
|---|---|---|
| SLI | A quantified measure of a user-facing service property | What proportion of valid lesson requests succeeded? |
| SLO | An agreed target for an SLI over a time window | What reliability should the course experience meet? |
| Error budget | The allowed unreliability implied by an SLO | How much budget remains before release risk needs review? |
| SLA | An external commitment with defined terms and consequences | What has the service promised to a customer? |
Page for Actionable User Impact
An alert should identify a symptom, explain its impact, and point to a next action. Page when a human needs to respond soon—for example, sustained user-facing failures or a rapidly exhausting error budget. Send lower urgency capacity trends and non-urgent maintenance to a ticket or dashboard. Alerts based on transient noise, missing labels, or an unowned service create fatigue and teach people to ignore the signal.
A practical on-call setup has an owner, a rota, a handoff, a runbook, escalation contacts, and a way to acknowledge and resolve incidents. During an incident, assess impact, communicate clearly, mitigate first when safe, and preserve a timeline. Afterward, review contributing conditions and improve tests, dashboards, automation, and the runbook without blaming individuals. Rotate fairly and provide backup so an alert is not dependent on one person being available.
Common Mistakes
Alerting on every small metric change
Why: Normal variance creates noise and responders stop trusting pages.
Fix: Use meaningful windows, user-impact signals, ownership, and an action for every page.
Putting request IDs or user IDs into metric labels
Why: Each unique value creates additional time series and uncontrolled cardinality.
Fix: Keep labels bounded; put request-specific context in redacted logs and traces.
Logging full request bodies and credentials to debug faster
Why: Secrets and personal data spread into a broadly accessible, retained system.
Fix: Log explicit safe fields, redact at the source, restrict access, and set retention intentionally.
Writing an SLO around a machine metric only
Why: Healthy CPU does not prove that a learner can load a lesson or submit practice.
Fix: Choose SLIs that measure an important user journey and define the eligible request population.
Instrument and Respond to a Service Signal
Practice
Create a useful view and a humane alert
- List one traffic, error, latency, and saturation metric for an API; note the source and units for each.
- Sketch a Grafana dashboard that helps an on-call engineer assess user impact and find the next diagnostic signal.
- Write an SLI and SLO for one learner journey, clearly defining valid requests and the measurement window.
- Draft an alert with a symptom, duration, severity, owner, runbook, dashboard, and immediate next action.
- Review a sample log event and remove credentials, payment data, and unnecessary personal identifiers.
Hint: Start with the user journey and the action a responder can take. A metric that cannot change a decision may not belong on the primary dashboard.
🎯 Key Takeaways
- ✓Metrics show trends and aggregate behavior, logs give event context, and traces connect work across service boundaries.
- ✓Instrument application behavior deliberately; Prometheus can scrape measurements but cannot invent useful business signals.
- ✓Use Grafana dashboards to answer operational questions and link from summary signals to logs, traces, and releases.
- ✓Define SLIs and SLOs around user journeys; an SLO, an error budget, and an external SLA are related but distinct.
- ✓Page on actionable user impact, not noisy fluctuations; give each alert an owner, runbook, and escalation path.
- ✓Keep sensitive data out of telemetry and avoid high-cardinality metric labels such as user or request IDs.
Next
What comes next
Define service level indicators and objectives, use error budgets to balance speed and reliability, and run incidents and blameless postmortems.
Lesson 14 · SRE Basics — SLIs, SLOs, Error Budgets and Incidents →HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
