Foundations · Lesson 04

Latency vs throughput

These two get swapped constantly in interviews. Latency is how long one request takes. Throughput is how many requests you handle per second. They're different axes — and improving one often does nothing for the other.

The pipe

Picture water moving through a pipe. Latency is how long a single drop takes to travel from one end to the other — set mostly by the pipe's length. Throughput is how much water exits per second — set mostly by the pipe's width. A long, fat pipe has high latency and high throughput. A short, thin straw has low latency and low throughput. They don't move together.

The practical punchline: adding servers usually buys throughput, not latency. Ten cashiers instead of one serves more shoppers per minute (throughput ↑) but each shopper's own checkout takes just as long (latency unchanged). To cut latency you shorten the path: move data closer (a CDN), cache it, remove a network hop, or do less work per request.

Don't measure with averages — use percentiles

Say 99 requests take 50 ms and one takes 5 seconds. The average is ~100 ms — which sounds fine and describes nobody's actual experience. That's why teams report percentiles:

• p50 (median): half of requests are faster than this. The typical experience.
• p99: 99% are faster; the slowest 1% are worse. This is the experience your unluckiest, often most valuable, users feel.

At scale the tail matters enormously: if a page makes 100 backend calls and each has a 1% chance of being slow, most page loads hit at least one slow call. Designing for p99, not the average, is a senior instinct.

Latency is the sum of four delays

"Latency" isn't one thing. A request's total time is the sum of four separate delays, and knowing which one dominates tells you exactly what to fix — speeding up the wrong one buys nothing.

Jitter is the variation in this total from one request to the next — mostly from changing queuing delay. Steady latency is easy to live with; jittery latency wrecks real-time apps like voice and video, which is why those buffer. The lever differs per delay: shrink payloads (transmission), move closer / use a CDN (propagation), add capacity or shed load (queuing), optimise code and queries (processing).

The numbers worth memorising

You don't need precision — you need the right order of magnitude so your back-of-envelope estimates land within 10×. These are the classic "latency numbers," rounded to remember:

The headline ratios: memory ≫ SSD ≫ disk, and a cross-continent network hop dwarfs almost any local work. That last fact is why caching and CDNs exist: serving from a nearby cache turns a 150 ms global round trip into a sub-millisecond local one.

Under the hood: measuring each delay with curl -w and reading a timing log

The four delays in the table above are not theoretical — they are individually measurable. curl's -w flag writes out timing variables after the transfer completes, one variable per phase. Here is the format string that maps directly to the four-delay model:

Each variable is cumulative from t=0, so to get the duration of each phase you subtract consecutive values:

Translating cumulative timestamps → per-phase durations:

DNS resolution:     0.018 s          ← time_namelookup
TCP connect:        0.067 − 0.018 = 0.049 s   ← propagation + queueing (one RTT)
TLS handshake:      0.119 − 0.067 = 0.052 s   ← TLS round trips (1-2 RTTs)
Server processing:  0.143 − 0.119 = 0.024 s   ← TTFB minus TLS done = server work
Transfer:           0.145 − 0.143 = 0.002 s   ← transmission delay (body was only 312 bytes)

Dominant cost here: TLS + TCP setup = ~101 ms (propagation/queueing)
Fix: keep-alive / connection reuse eliminates TCP+TLS on subsequent requests

Computing p50 / p99 from a log. Once you have a stream of response times (e.g. from an access log or a simple loop), percentiles are straightforward to compute. Here is the pattern using command-line tools:

That p99 gap is the "tail latency" problem in action. In production, use your APM or Prometheus histogram (e.g. histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))) instead of a shell loop — but the math is identical.

How to debug & inspect it — slow-endpoint triage

When an endpoint is slow, the four delays give you a structured triage: identify which phase is large, and only then look for fixes in that area. The table below maps symptoms to the delay type and the lever to pull.

Slow-endpoint triage checklist:

1. Run curl -w with the timing format above (or pull from APM) and compute per-phase durations.
2. Identify the dominant phase — that is the only one worth optimising. Fixing the wrong phase wastes effort.
3. If processing (TTFB) is large: add tracing inside the endpoint to find the slow DB call or computation.
4. If connect or TLS is large: the fix is geographic (CDN/edge), not code.
5. Collect p50 and p99. A high p99/p50 ratio (> 3×) almost always indicates queuing — look for thread-pool or DB-pool exhaustion.
6. Always re-measure after each change to confirm the phase shrank, not just the total.

By the numbers

Concrete scenario: a checkout API handles 5,000 req/s with a mean service time W = 40 ms (0.04 s). A fan-out page loader calls N = 100 microservices per page, each with a p = 1% chance of hitting the slow tail (>500 ms).

Little's Law — how many requests are in flight right now?

Little's Law is the single most useful identity in queuing theory. For any stable system:

For our checkout API: L = 5,000 × 0.04 = 200 concurrent requests in-flight at steady state. That is how many threads, goroutines, or connection-pool slots must be available just to keep up. If you size the thread pool at 150, you are already 25% short at today's load.

Queueing blow-up as utilisation ρ approaches 1

Utilisation ρ = λ / μ, where μ is the server's maximum throughput (requests per second it can handle). In an M/M/1 model the average wait in queue is:

The wait multiplier ρ / (1 − ρ) blows up near ρ = 1 — this is why "90% utilised feels fine but 99% utilised is catastrophic":

At ρ = 0.9 the wait alone is 9× the service time. The queue is not a linear function — it is a hyperbola that goes vertical. The practical rule: target ρ ≤ 0.7 for latency-sensitive paths so a 2× traffic spike only pushes you to ρ = 0.85 rather than over the cliff.

Fan-out tail: P(at least one slow call) = 1 − (1 − p)^N

A page that fans out to N independent services, each with a p = 1% chance of being slow on any given call:

Trace for p = 0.01 (1% per service):

At N = 100 services per page, a 1% per-service tail hits 63% of all page loads. That is why a p50 of 40 ms can coexist with a p99 of 800 ms — the average is fine; the fan-out exposes the tail on almost every complex request. (Dean & Barroso, "The Tail at Scale," CACM 2013)

Worked trace — a burst at 08:15:00 UTC

Scenario: the checkout fleet processes at μ = 6,000 req/s max capacity. A flash sale fires up at 08:15:00:

The key lesson from the trace: by the time ρ = 0.95 is visible in metrics, you are already 30 seconds from a full queue collapse. You need to scale at the 0.83 alarm, not at the timeout.

Decision math — how much headroom do you need?

To guarantee p99 ≤ some target latency T under a 2× traffic spike, work backwards:

Translation: if your flash sales can double traffic, run at ≤ 40% utilisation in steady state. At μ = 6,000 req/s that means planning for a maximum of 6,000 × 0.40 = 2,400 req/s steady-state before you start provisioning more capacity. Equivalently, size the fleet so that peak-day traffic uses no more than half the installed capacity — a rule of thumb used by Google SRE (Google SRE Book, Chapter 22).

Sources & further reading

• "Latency Numbers Every Programmer Should Know" — interactive (after Jeff Dean)
• Cloudflare — What is latency?
• MDN — Understanding latency