Reliability & Scale · Lesson 07

Caching at Every Layer

A cache is the closest thing to a free lunch in distributed systems — the same byte of data, served a thousand times, need only be fetched once. But caches are only free when you get invalidation right, and that turns out to be one of the genuinely hard problems in computer science.

Why caching exists: two problems at once

Every request that reaches your database costs money in CPU, I/O, and time. Popular data is read far more than it is written — a product page, a user profile, a configuration value. The 80/20 rule applies with brutal consistency: 20% of your data accounts for 80% of your reads. A cache exploits this asymmetry by keeping the popular 20% in fast, cheap memory so the database never sees most requests at all.

The analogy: a reference librarian who gets asked the same twenty questions every day eventually keeps a cheat sheet on her desk. The first asker went to the stacks; everyone after that gets an instant answer from the desk. The database is the stacks. The cache is the cheat sheet.

Two distinct problems are solved simultaneously:

• Latency — memory is nanoseconds; disk and network are milliseconds. A cache hit is 100–10 000× faster than a cache miss.
• Load — a database with 1 000 requests/s becomes a database with 50 requests/s if the hit rate is 95%. The database handles more total traffic at lower peak load.

The five layers of caching

Caching is not a single thing — it is a stack of independent stores, each closer to the user than the last. A request that misses every cache falls all the way to the database; one that hits the first layer never touches the network.

HTTP caching: Cache-Control, ETag, and 304

HTTP has a built-in caching protocol that browsers, CDNs, and reverse proxies all understand. The core mechanism is the Cache-Control response header.

# Response from an API endpoint
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: public, max-age=300, stale-while-revalidate=60
ETag: "v3-a1b2c3d4"
Last-Modified: Thu, 19 Jun 2026 08:00:00 GMT
Vary: Accept-Encoding

{ "product_id": 9, "name": "Titanium Frame", "price": 499 }

# Client re-requests 6 minutes later (past max-age=300)
GET /v1/products/9 HTTP/1.1
If-None-Match: "v3-a1b2c3d4"

# Server — data unchanged — responds without a body
HTTP/1.1 304 Not Modified
Cache-Control: public, max-age=300, stale-while-revalidate=60
ETag: "v3-a1b2c3d4"

Key directives:

The ETag header is a version token for the resource — a hash, a version number, or any opaque string that changes whenever the data changes. On subsequent requests the client sends If-None-Match: "<etag>". The server compares this to the current ETag. If unchanged, it returns 304 Not Modified with no body — saving the bandwidth of the full response. If changed, it returns 200 OK with the new body and a new ETag.

The bandwidth saving is significant: a 304 response carries only headers, not the payload. A 50 KB JSON response becomes a few hundred bytes on a cache hit.

Write strategies: getting data into the cache

Reading from a cache is simple. The hard question is when and how does the cache get updated after a write? There are three main strategies:

Cache-aside (lazy loading)

The application is responsible for the cache. On a read, it checks the cache; if it misses, it reads from the database and populates the cache. On a write, it writes to the database and then invalidates (deletes) the cache entry, so the next read re-populates it with fresh data.

Pros: only data that is actually read gets cached — no wasted memory for cold data. Easy to implement. Cons: the first read after a write always hits the database (cache miss). Two operations (DB write + cache delete) are not atomic; a failed invalidation leaves stale data.

Write-through

On every write, the application writes to both the cache and the database synchronously. Reads always hit the cache; it is always up-to-date. Cons: every write pays the overhead of two stores. Data that is never read is still cached, wasting memory.

Write-back (write-behind)

Writes go to the cache immediately; the database is updated asynchronously in a background process. Very fast write latency. Cons: if the cache node fails before the DB write commits, data is lost. Complex to implement correctly. Use primarily for high-throughput counters, analytics aggregations, and other loss-tolerant workloads.

Invalidation: the genuinely hard problem

Phil Karlton's famous quip — "There are only two hard things in computer science: cache invalidation and naming things" — is funny because it is true. A cache that holds data nobody reads is harmless. A cache that serves wrong data to users is a production incident.

The three main invalidation approaches:

• TTL (Time-To-Live) — every cache entry has a maximum age. After it expires the entry is evicted and the next read goes to the source of truth. Simple to implement; accepts a bounded window of staleness. The TTL is a trade-off knob: short TTL = fresher data, higher load; long TTL = more load reduction, higher staleness risk.
• Explicit invalidation — on every write, delete the corresponding cache key. Guarantees freshness immediately after a write. Fails silently if the cache delete fails (retry with care; delete is idempotent). Fragile when one logical object is cached at multiple keys or derived from multiple tables.
• Event-driven invalidation — write to the database, publish a "resource updated" event, cache subscribers listen and evict the relevant keys. Decoupled and scalable; adds complexity of an event bus and eventual consistency lag.

Cache stampede (thundering herd on expiry)

A cache stampede occurs when a popular cached item expires and dozens of concurrent requests all miss simultaneously, all query the database at once, all compute the same answer, and all try to write it back to the cache. The database experiences a sudden spike for exactly the duration of one slow query.

Mitigations:

• Probabilistic early expiry (also called "jitter on TTL"): refresh the cache slightly before it expires, with probability proportional to how close it is to expiry. No thundering herd because one request gets ahead of the stampede.
• Mutex / lock: the first miss acquires a lock; subsequent misses return the stale value (or a loading indicator) while the lock-holder recomputes. Only one database query per stampede event.
• Background refresh: a background job pre-warms popular keys before they expire, so the cache is never empty.

Under the hood: how the HTTP cache decision actually works

When the browser (or a CDN) receives a response, it runs a deterministic decision algorithm before touching the network again. Here is that algorithm traced end-to-end for a single resource.

Step 1 — Parse Cache-Control and compute the freshness lifetime

The cache reads the directives in order of precedence: s-maxage (shared caches only) > max-age > Expires header > heuristic freshness (typically 10% of Last-Modified age, capped). The result is a single integer: the freshness lifetime in seconds.

Step 2 — Serve from cache while fresh

On every subsequent request, the cache computes current age = now − stored_at + Age_header_value. If current_age < freshness_lifetime, the response is fresh: serve it immediately, no network. The server sees zero bytes.

Step 3 — On expiry, revalidate with If-None-Match

At t=301 s the cached copy becomes stale. Rather than fetching the full body again, the cache sends a conditional GET using the stored ETag:

The cache resets its freshness timer and continues serving from its copy. If the product had changed, the server returns 200 OK with the new body and a new ETag, and the cache replaces its stored copy.

The stale-while-revalidate grace window

Between t=300 and t=360 the entry is stale-but-within-grace. The cache serves the stale copy immediately (zero latency impact) and fires the conditional GET in the background. The user sees no delay; the next request (after the background fetch completes) gets the fresh copy. This is how high-traffic APIs can have a 5-minute cache but still feel instant during revalidation.

Cache key and Vary

The cache key is not just the URL. A Vary header tells the cache which request headers are part of the key. Vary: Accept-Encoding means the cache stores separate entries for gzip-compressed and uncompressed responses to the same URL. Vary: Authorization would be catastrophic — every user would get their own entry in a shared cache, defeating the purpose. The cache key algorithm is:

cache_key = method + url + SORTED(vary_header_values_from_request)
# e.g.:  GET:/v1/products/9:gzip   ≠   GET:/v1/products/9:identity

How to debug & inspect caching

The fastest tool is curl -I (HEAD request) or curl -I --head to read only response headers. Add -H "Cache-Control: no-cache" to force a fresh fetch and compare the headers before and after TTL expiry.

In Chrome DevTools: open Network, click the request, look at the Response Headers tab. The x-cache header (CDN) and age header reveal whether the CDN served the response and how old it was. A "(disk cache)" label in the Size column means the browser served it without any network request at all.

Debug checklist for "why isn't this cached?"

1. Run curl -sI <url> and read Cache-Control, Vary, ETag, x-cache, and age headers.
2. Check for Set-Cookie on the response — most CDNs bypass caching whenever a cookie is set.
3. Confirm the public directive is present and no-store / private are absent.
4. If caching per content-type (gzip vs plain), verify Vary: Accept-Encoding is set.
5. For "stale after deploy": purge the CDN and verify the ETag changed in the new response.

By the numbers

Scenario: a product catalogue API at 10 000 req/s. Redis cache latency L_cache = 1 ms; origin (PostgreSQL) latency L_origin = 50 ms. Current hit ratio h = 0.90 (90% of requests served from cache).

Effective latency formula

effective_latency = h × L_cache + (1 − h) × L_origin

At h = 0.90:

Hit ratio table: latency and origin load

Origin load cut = fraction of requests that reach the database = (1 − h). At 10 000 req/s baseline:

The origin QPS column shows why caching is a load-reduction tool as much as a latency tool: at h=0.99 the database sees only 100 req/s instead of 10 000. Sources: AWS — Caching best practices; Redis latency benchmarks: Redis — Benchmarks.

Decision math: cache size vs. working set to reach a target hit ratio

Cache hit ratio is dominated by whether the working set (the hot keys actually requested) fits in the cache. By Zipf’s law, the top-20% of keys account for ~80% of traffic. To reach h = 0.90, you need to cover the top-10% of keys by request frequency.

Worked example: the catalogue has 1 000 000 products. Average cached value size = 2 KB. To cover the top 10% of keys:

This means for most read-heavy workloads with repeated keys, an h of 0.90–0.99 is achievable with a cache that is 2–20% of total data size. The formula for the minimum cache entries needed to reach hit ratio h under a Zipf working set:

min_keys_cached = total_keys × (1 − h)   # top (1−h) fraction by frequency
cache_bytes     = min_keys_cached × avg_value_size

When caching pays off

Caching is worth its operational cost when both conditions hold:

Break-even: caching pays off when h × L_origin > L_cache + invalidation_overhead. With L_origin = 50 ms, L_cache = 1 ms, and negligible invalidation overhead: break-even at h > 1/50 = 2%. Any hit ratio above 2% makes caching faster than going to origin every time — which is why even a small in-process LRU cache is almost always worthwhile for read-heavy endpoints.

Multi-tenant: the cache key must include the tenant

The single most common caching security bug in multi-tenant APIs is the cross-tenant cache leak: a per-user response is cached under a key that does not include the user's identity, and a different user's request gets a cache hit — receiving the first user's private data. This is not a theoretical concern; it surfaces regularly in production systems where caching is added to an existing authenticated endpoint without adjusting the key scheme.

The bug: caching GET /v1/me under the path alone

Two users — uA (Alice, user_id: 1001) and uB (Bob, user_id: 2055) — both call GET /v1/me to fetch their own profiles. A naive Redis cache stores the response under the key "cache:GET:/v1/me" — no user identity, just method + path.

The fix: key = (user_id, method, path [, relevant headers])

The corrected key scheme binds the cache entry to the tenant's identity extracted from the authenticated context (never from a user-supplied value in the URL):

# Correct: derive user identity from the validated token, not the path
function cache_key(user_id, method, path, vary_headers):
  base = method + ":" + path
  return "cache:" + user_id + ":" + base
  # e.g. "cache:1001:GET:/v1/me"  and  "cache:2055:GET:/v1/me"
  # → completely separate entries; uB can never read uA's slot

Corrected trace with the same two users:

The Vary header and Cache-Control: private for shared caches

The same isolation principle applies at HTTP caches (CDNs, reverse proxies). Two tools enforce it:

• Cache-Control: private — instructs any shared cache (CDN, proxy) that this response must not be stored at all; only the end-user's own browser may cache it. This is the correct directive for any response containing per-user data. Adding private is simpler than configuring a shared cache to key by identity.
• Vary: Authorization — if you do need a shared cache to store per-token responses (unusual), the Vary header tells the CDN to treat each distinct value of the Authorization header as a separate cache key. In practice most CDN vendors either ignore Vary: Authorization or cache the first value and serve it to everyone — making Cache-Control: private the far safer choice. See MDN — Vary header and OWASP — Cache Poisoning / Web Messaging for attack scenarios.

By the numbers: the cost of per-tenant key isolation

With 50,000 active users each having a separate cache entry for GET /v1/me, and an average profile size of 1 KB:

Sources & further reading

• MDN — HTTP caching
• MDN — Cache-Control header
• MDN — ETag header
• Cloudflare — What is caching?
• AWS — Caching best practices