API Design

Production at Scale · Simulator 06

Capacity & the latency hockey-stick

Why do engineers keep servers at 60–70% utilization and refuse to "just run them hotter"? The M/M/1 queueing model answers it: latency rises hyperbolically as utilization approaches 100%, producing the famous hockey-stick curve. Drag QPS up, watch the dot climb the curve — and watch the cost.

InteractiveDrag the slidersModels rel-14, rel-13
💡 How to Read the Simulator

Curve: response time vs utilization (M/M/1). Current operating point shown as a dot (●). The vertical dashed line marks ρ = target utilization. Notice how small increases in ρ near 1.0 produce massive latency increases.

By the numbers: capacity and queueing math

Given a target utilization u, you need enough servers so no single server is overloaded. The M/M/1 model (single-queue, exponential service) gives the mean response time:

# Minimum servers to keep utilization ≤ target
servers      = ceil(QPS / (perServer × targetUtil / 100))

# Actual utilization with that fleet size
ρ            = QPS / (servers × perServer)

# M/M/1 mean response time (service time + queueing delay)
responseTime = serviceMs / (1 − ρ)           # diverges as ρ → 1

# Illustrative monthly cost ($0.10/hr per server, 730 hr/month)
monthlyCost  = servers × $0.10 × 730

The key insight: responseTime = serviceMs / (1 − ρ) has a singularity at ρ = 1. At 50% utilization you pay 2× the service time; at 90% you pay 10×; at 99% you pay 100×. That is why keeping utilization at ≤ 70% is not waste — it is the headroom that absorbs traffic spikes without blowing latency SLOs.

Utilization vs Latency worked trace

Utilization (ρ) Service Time (ms) Queueing Delay (ms) Total Latency (ms) Slo Budget Consumed (of 50ms)
10% 20.0 ms 2.2 ms 22.2 ms 44.4%
50% 20.0 ms 20.0 ms 40.0 ms 80.0%
80% 20.0 ms 80.0 ms 100.0 ms 200.0% (BREACHED)
95% 20.0 ms 380.0 ms 400.0 ms 800.0% (CRITICAL)
✅ Try this

1. Set target utilization to 95% — notice how the response time blows up near that operating point and how little headroom you have for spikes. 2. Drop it back to 65% — you need more servers and it costs more, but the operating point sits in the flat part of the curve, well away from the knee. 3. Raise QPS to 100M with a small per-server capacity — watch servers and cost scale up. 4. Raise Per-server capacity (vertical scaling) and watch Servers needed and cost fall for the same QPS and target — the horizontal alternative is simply provisioning that derived server count. Compare the cost change.

⚠️ Modeled, not measured

This uses the M/M/1 queueing approximation (Poisson arrivals, exponential service, single queue). Real systems have multiple queues, non-exponential service times, connection limits, and GC pauses that change the shape of the curve — but the qualitative behaviour (latency diverges as ρ → 1) holds universally. The $0.10/hr cost is purely illustrative. Treat numbers as directional, not operational.

Under the hood: how CPU saturation & response time curves actually work

The knee itself is a queueing-theory result, not an OS artifact: mean wait W_q = S · ρ/(1−ρ) diverges as ρ→1 regardless of the scheduler (that is the formula the table above is generated from). On top of that, at very high CPU utilization the OS scheduler adds a second, compounding source of delay. That amplifier works as follows:

  1. Request Arrival: The network card places incoming packets into a kernel-space TCP backlog ring buffer.
  2. Process Scheduling: The kernel's completely fair scheduler (CFS) allocates time slices to application worker threads. When CPU utilization crosses 85-90%, threads spend more time waiting in the run queue (runnable status) than actually executing: 
    # OS Scheduler Task State: Runnable → Running
Thread-1: [   RUNNING   ][ WAIT ][   RUNNING   ]
Thread-2: [ WAIT ][   RUNNING   ][ WAIT ][ RUN ]
  3. Backlog Overflow: As threads stall in the scheduling queue, they pull requests slower from the TCP socket buffer. The buffer fills, resulting in packet drops and client-side connection timeouts.
🎯 Interview angle

Expect to derive, not recite:

How to debug & inspect it

To audit and troubleshoot capacity constraints, inspect OS scheduler statistics and check network socket backlog depths on the server nodes.

$ ss -lnt '( sport = :8080 )' State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 129 128 *:8080 *:* # Recv-Q > Send-Q indicates the application socket backlog is overflowing

Symptom → Cause → Fix:

Symptom Likely Cause Fix
Request latency rises exponentially while CPU utilization is near 90%+ CPU core starvation causing scheduler queue delays (M/M/1 knee) Autoscale the server fleet horizontally or configure load shedding to fail fast
Connection timeouts (504 Gateway Timeout) while CPU is low (<15%) TCP listen backlog is full because thread-pool sizes are set too low Increase application thread-pool limits or tune kernel parameter net.core.somaxconn
Servers run out of memory (OOM crash) during sudden load spikes Unbounded in-memory request buffering consumes all virtual memory Enforce queue size limits and return HTTP 503 Service Unavailable when full

Sources & further reading