Reliability & Scale · Lesson 09

Monitoring & Observability

An API you cannot measure is an API you cannot trust. Observability is the engineering discipline of making a system's internal state inferable from the signals it emits — so you can answer "is it healthy?" without logging into the box to look.

The dashboard that lies to you

A server that responds with HTTP 200 to every request can still be quietly wrong: returning stale data, taking 8 seconds per call, silently dropping one in fifty writes. A CPU graph at 30% doesn't catch any of that. Monitoring that shows only "server up / server down" is a smoke detector with no batteries — it passes every test until the house is on fire.

Observability is the practice of instrumenting a system so you can ask any question about its behaviour and get a real answer. The three complementary tools that make this possible are metrics, logs, and traces — the "three pillars."

The three pillars in depth

Metrics are numeric measurements sampled over time: request rate (req/s), p99 latency (ms), error count, CPU %. They're cheap to store as aggregates, easy to graph and alert on, and excellent for spotting trends. They cannot tell you why something is wrong — just that something is.

Logs are discrete, timestamped records of events: "user 42 attempted login, result: bad password." They contain rich context — user ID, request body, stack traces — and are invaluable for debugging specific incidents. The cost is storage: a busy service can emit gigabytes of logs per hour, so selective retention and structured formatting matter.

Traces follow a single request as it crosses service boundaries — from the edge load balancer through the API server to the database and back. Each hop produces a span with a start time and duration. The tree of spans is a trace. Traces reveal exactly which step is slow in a chain of microservice calls — the thing metrics and logs cannot show directly.

The four golden signals

Google's SRE book distils everything you need to measure about a user-facing service into four signals. If you only have time to instrument four things, instrument these:

RED and USE: two practical checklists

Two complementary methods reduce "what should I monitor?" to a short checklist:

• RED (for services / APIs): Rate, Errors, Duration. A microservice is healthy if its request rate is normal, its error rate is low, and its response durations are within SLO.
• USE (for resources — CPU, disks, queues): Utilisation, Saturation, Errors. A database connection pool is healthy if utilisation is under ~70%, the queue waiting for a connection (saturation) is near zero, and no connection errors occur.

Apply RED to every service endpoint; apply USE to every shared resource underneath it. Together they cover almost everything that causes incidents.

SLO-driven alerting

An alert that fires every time a metric crosses a hard threshold ("alert if p99 > 200 ms") is a recipe for alert fatigue — it fires during routine traffic spikes, wakes engineers unnecessarily, and trains people to ignore it. SLO-driven alerting inverts the question: instead of "is this metric unusual?" it asks "are we on track to exhaust this month's error budget?" (see Lesson 10 on SLOs).

The practical pattern is the burn-rate alert. If your SLO is 99.9% availability over 30 days, your error budget is 43.2 minutes of downtime. A burn rate of 1× means you'd exhaust it in exactly 30 days. A burn rate of 14× means you'd exhaust it in ~52 hours. Alerting on a high burn rate (e.g. >14× for 5 minutes) catches serious degradations quickly; a low sustained burn rate (e.g. >3× for 30 minutes) catches slow erosion before the budget disappears.

Distributed tracing: traces, spans, and correlation IDs

When a single user action travels through five microservices, a stack trace from service 3 alone isn't enough — you need the whole journey. Distributed tracing reconstructs that journey by attaching a trace ID to the incoming request and propagating it in every outbound call as an HTTP header (traceparent in the W3C standard, X-B3-TraceId in Zipkin). Each service creates a span and reports it (with the trace ID, a parent span ID, timing, and metadata) to a tracing back-end like Jaeger or Tempo.

A correlation ID (also called a request ID) is a simpler version of the same idea for monoliths or two-service architectures: generate a UUID at the edge, pass it in a header (X-Request-Id), and emit it in every log line. Now you can grep all logs for one ID and reconstruct the full lifecycle of a single request across log files.

Structured logging

Unstructured logs are English sentences: "User 42 logged in from 1.2.3.4 at 2025-09-01 10:00". Parsing that with regex is fragile. Structured logs emit machine-readable key-value records (JSON being the standard) so log aggregation platforms can filter, group, and alert on any field without text parsing:

{
  "timestamp": "2025-09-01T10:00:00Z",
  "level":     "INFO",
  "service":   "auth-service",
  "event":     "user.login",
  "user_id":   42,
  "ip":        "1.2.3.4",
  "trace_id":  "f3a8b9c1d2e3",
  "duration_ms": 58
}

Every field is queryable. Finding all slow logins from a specific IP is now level=INFO AND event=user.login AND duration_ms > 500 AND ip=1.2.3.4 rather than a grep pipeline.

Under the hood: how telemetry is actually collected

Counters vs histograms — and where p99 comes from

Prometheus (the de-facto standard) has four metric types. The two you see most often in API monitoring are Counter and Histogram.

A Counter is a monotonically increasing integer. You increment it when something happens: a request completes, an error fires, a payment is processed. The raw value is meaningless on its own — what matters is its rate of change. PromQL's rate() function computes how many increments per second occurred over a sliding window:

A Histogram tracks value distributions — request durations, payload sizes. Instead of storing every measurement, it increments a set of pre-defined buckets. A request of 87 ms increments every bucket whose upper bound is >= 87 ms:

The key insight: averages hide tail latency. A mean of 50 ms can coexist with a p99 of 2 000 ms if a small fraction of requests hit a slow code path. Histograms make the tail visible. Choose bucket boundaries that straddle your SLO threshold (if your SLO is 99% of requests under 300 ms, put a bucket at 0.3).

Trace context propagation — how a traceparent header carries a trace across services

The W3C Trace Context spec defines a header, traceparent, that encodes four fields as a hyphen-separated string:

Here is what happens when a request crosses three services — from creation to span assembly in the tracing back-end:

Every service only needs to: (1) read the incoming traceparent header; (2) create a child span using that trace-id and the received span-id as its parent; (3) propagate the header outbound with its own span-id as the new parent. OpenTelemetry auto-instruments most frameworks to do all three automatically.

How to debug & inspect monitoring

PromQL availability + p99 queries

Reading a trace waterfall to find the slow span

Open the trace in Jaeger, Tempo, or Honeycomb. Sort spans by duration descending. The widest child span is your bottleneck. The gap between spans (white space) is time the parent spent computing locally — narrow gaps and a wide child mean the child is the problem; wide gaps and narrow children mean the parent's own code is slow.

Using correlation/request IDs to stitch logs

If distributed tracing is not yet set up, a request ID in every log line provides the same cross-service join:

Debug checklist for "p99 is high, where is it?"

1. Run histogram_quantile(0.99, ...) broken out by handler — identify which endpoint is slow.
2. Open a trace for a slow request in Jaeger/Tempo — sort child spans by duration.
3. Find the widest span; check the gap before and after it for hidden local computation.
4. Grep service logs by trace_id or request_id to get the log-level context around the slow span.
5. If no trace is available: add a span around the suspected code path; re-deploy; re-measure.

By the numbers

Make the math concrete. Scenario: a payment API at 5,000 req/s, 1% trace sampling, Prometheus histograms with 6 latency buckets.

p99 via linear interpolation from histogram buckets

Prometheus histograms store cumulative counts per bucket. To find the p99 you locate the bucket that straddles the 99th-percentile rank and linearly interpolate within it.

This is the value histogram_quantile(0.99, ...) returns. The result is an approximation — the true p99 lives somewhere in [250 ms, 500 ms]. Bucket boundaries are the knobs: if your SLO is "99% of requests under 300 ms," add a bucket at le=0.3 to bracket the threshold precisely. (Prometheus histogram best practices)

Counter rate formula

Sampling: how many traces does 1% capture?

At 5,000 req/s and 1% head sampling:

Is 50 traces/s enough to compute p99 from traces? No — p99 requires a representative sample of the tail. With 50 traces/s sampled uniformly, you expect to capture roughly 50 × 0.01 = 0.5 of the slowest 1% per second — less than one slow trace per second. Histograms (which measure all 5,000 req/s) are better for p99; traces are better for diagnosing why a specific slow request was slow.

Decision math — sampling rate vs storage: at 1% sampling and 10 KB/trace, daily storage is ~43 GB, which is reasonable for most teams. To actually debug tail latency, use tail-based sampling (keep 100% of requests that took >500 ms, drop the fast ones) rather than head-based sampling. Tail sampling keeps the traces that matter and controls storage. (OpenTelemetry — Sampling concepts)

Decision math: bucket boundaries for useful p99 resolution

If your SLO threshold is 300 ms and your buckets jump from 250 ms to 500 ms, you cannot tell whether p99 is 260 ms (well inside SLO) or 490 ms (nearly double the SLO). The interpolation uncertainty is the entire bucket width. Add boundaries that straddle your SLO:

Sources & further reading

• Google SRE Book — Monitoring Distributed Systems (four golden signals)
• OpenTelemetry — Observability primer (metrics, logs, traces)
• Google SRE Workbook — Alerting on SLOs (burn-rate approach)
• Brendan Gregg — The USE Method