Reliability & Scale · Lesson 02

Idempotency

Networks drop packets; clients time out; retries happen. Idempotency is the property that makes retrying safe — so the second (and third, and tenth) attempt produces the same result as the first, with no extra side effects piling up.

What idempotent means, exactly

The word comes from Latin: idem (same) + potens (power). In mathematics, a function is idempotent if applying it multiple times has the same effect as applying it once: f(f(x)) = f(x). In HTTP, an operation is idempotent if the server-side state ends up identical whether you sent the request once or a hundred times.

The classic light-switch analogy breaks down here because "toggle" is not idempotent — each press changes state. A better analogy is a thermostat set-point: telling your thermostat "set temperature to 22°C" ten times in a row ends in exactly the same state as telling it once. The action is about declaring a desired state, not incrementing a counter.

Which HTTP methods are idempotent?

The HTTP specification (RFC 9110) is explicit about this:

The non-idempotency of POST is not a design flaw — it is the point. POST /orders is supposed to create a new order each time. The problem arises when a network failure leaves the client uncertain whether the request arrived: retrying may create a duplicate. That's the gap idempotency keys fill.

Why retries demand idempotency

Distributed systems have exactly two outcomes after sending a request: the server processed it, or it didn't. But the client often cannot tell which. A TCP timeout can happen after the server committed the transaction but before the response bytes arrived. The client sees only silence. Its two choices are:

1. Give up — safe, but now the user's payment or order may be silently lost.
2. Retry — recovers from transient failures, but risks a duplicate if the first attempt succeeded.

Neither choice is acceptable on its own. Idempotency keys resolve the dilemma: retry freely, and let the server figure out whether this is genuinely new work or a duplicate it already handled.

Idempotency keys: how they work

The pattern is straightforward:

1. The client generates a unique key (a UUID v4 is typical) for this logical operation and attaches it to every attempt of the same request.
2. The server stores the key in a deduplication store (Redis, a DB table) the first time it sees it, along with the operation's final response.
3. On a retry, the server looks up the key, finds the stored response, and returns it directly — no second execution.
4. The key expires after a reasonable window (24 hours for payments is common), after which a new request with the same key is treated as a new operation.

# First attempt — client generates a stable UUID for this charge
POST /v1/charges HTTP/1.1
Host: api.payments.example
Authorization: Bearer sk_live_abc123
Idempotency-Key: a3f8c2d1-7b4e-4a9f-b3c2-1d8e9f0a2b4c
Content-Type: application/json

{
  "amount": 4999,
  "currency": "usd",
  "source": "tok_visa",
  "description": "Order #8812"
}

# Server response — charge created, stored against the key
HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": "ch_3Pz7kL2eZvKYlo2C1j8mR4nQ",
  "amount": 4999,
  "status": "succeeded"
}

Now suppose the response never reached the client. On retry:

At-least-once delivery and why it pairs with idempotency

Distributed messaging systems (queues, webhooks, event buses) guarantee at-least-once delivery: a message will eventually reach a consumer, possibly more than once. "Exactly-once" is mathematically hard to guarantee end-to-end across failures. The practical solution is to design consumers to be idempotent — receiving a duplicate message a second time changes nothing. Together: at-least-once delivery + idempotent consumers = effectively-once semantics from the perspective of state.

Under the hood: how an idempotency-key store actually works

The phrase "store the key and return the response on retry" hides several important implementation details. Here is the exact mechanism, the state the store carries, and the race conditions you must handle.

The atomic insert-if-absent

The critical primitive is insert the key only if it does not already exist, atomically. In Redis this is SET key value NX (NX = "only if Not eXists"). In a SQL database it is an INSERT … ON CONFLICT DO NOTHING inside a transaction, or a unique constraint on the key column.

# Redis — atomic insert-if-absent
# Returns OK if the key was new, nil if it already existed
SET ik:a3f8c2d1-7b4e-4a9f-b3c2-1d8e9f0a2b4c "pending" NX EX 86400
#                                                 ^^        ^^ TTL: 24 h
#   NX = only set if not exists

If two concurrent retries both reach Redis at the same instant, exactly one will receive OK and the other will receive nil — Redis is single-threaded for command execution, so there is no ambiguity. The winner proceeds to execute the business logic; the loser waits (see below).

State table: what the store holds

Concrete flow traced end-to-end

Handling a concurrent in-flight duplicate

The tricky case: two retries arrive simultaneously, neither has stored a final response yet, and the key holds "pending". Options:

1. Return 409 immediately — "a request with this key is currently in-flight; wait and retry." Simple, but requires the client to handle 409.
2. Short poll — wait 50–200 ms, re-check the key; if it transitions from "pending" to a final response within a short window, return it. Best for latency-sensitive clients.
3. Use a distributed lock — acquire a lock on the key before executing; the second concurrent arrival blocks until the lock is released. More complex but avoids the 409 round-trip.

Key TTL and expiry policy

A 24-hour TTL is the Stripe standard for payment idempotency keys. The tradeoff: too short and a legitimate slow retry (hours later, due to a batch job) creates a duplicate; too long and the deduplication table grows without bound. Choose the TTL to match your retry window — the maximum time a client could reasonably retry the same operation.

-- SQL alternative: dedup table with partial index for active keys
CREATE TABLE idempotency_keys (
  key          TEXT        PRIMARY KEY,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  status       TEXT        NOT NULL,   -- 'pending' | 'complete'
  response     JSONB                     -- null while pending
);
-- Insert atomically inside the same DB transaction as the business record
BEGIN;
  INSERT INTO idempotency_keys(key, status)
    VALUES('a3f8c2d1-…', 'pending')
    ON CONFLICT(key) DO NOTHING;
  -- IF 0 rows inserted, key existed → roll back, return stored response
  INSERT INTO charges(...) VALUES(...);
COMMIT;

Inserting the idempotency key and the business record in the same transaction means they either both commit or both roll back. This prevents the failure mode where the business record is written but the key write fails, leaving no deduplication guard for retries.

How to debug & inspect it

Idempotency bugs usually manifest as either duplicate side effects (a key was not stored correctly, so retries re-execute) or phantom 409s (a key is permanently stuck in "pending" due to a crash mid-execution). Both are diagnosable by replaying requests and reading the dedup store directly.

Test by replaying the same key

Symptom → cause → fix

1. In staging, always run a replay test after any change to the idempotency path: send the same key twice and assert the side-effect count (charges, rows) is exactly one.
2. In production logs, correlate Idempotency-Key with the charge/order ID to confirm unique keys produce unique resources.
3. Monitor the 409 rate on idempotency-key endpoints — a spike indicates a stuck-pending problem or a client bug re-using keys.
4. Check the dedup store TTL on every key write; a missing EXPIRE means keys accumulate forever.

In production: how leading APIs do it

Money-handling APIs converged on a remarkably consistent design for idempotency — not by coordination, but because every team that built payment infrastructure at scale hit the same failure modes and arrived at the same solution independently.

Why this exact design is the industry standard. Every payment API above independently arrived at the same three-part design: a client-supplied key, a server-side deduplicated result store, and a fixed expiry window. The convergence is not accidental — it reflects the hard constraints of the problem.

First, the key must be client-supplied. The server cannot generate a deduplication key because the key must be stable across network failures — the client must be able to resend the exact same key on retry, before it knows whether the first attempt succeeded. A server-generated key would require a successful round-trip to obtain, which is exactly the situation idempotency is designed to handle.

Second, the key must be stored with the result, not just the fact of execution. Storing only "this key was processed" and then re-executing to build the response on replay would introduce a second execution window. The stored response must be verbatim so that replays are indistinguishable from the original — same charge ID, same status, same body.

Third, the expiry window (24–72 hours across providers) reflects the practical retry window for automated systems. A batch job running daily might retry a failed payment up to 24 hours later; a retry window shorter than that would treat a legitimate late retry as a new operation and risk a duplicate. A window significantly longer than the maximum realistic retry interval wastes storage on keys that will never be replayed.

The result: client-supplied UUID + server-side result store + expiry window is the minimal correct design for safe retries on non-idempotent endpoints. Every variation across providers (header vs. body field, 24h vs. 72h, response on key conflict) is a parameter choice, not a structural difference.

How leading APIs do it

-- Pseudo-schema for the dedup table
CREATE TABLE idempotency_keys (
  key          TEXT PRIMARY KEY,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  status       TEXT,        -- 'pending' | 'complete'
  response     JSONB        -- stored full response
);

Sources & further reading

• Stripe — Idempotent Requests — Stripe's production-proven explanation and implementation of idempotency keys
• IETF Draft — The Idempotency-Key HTTP Header Field — working draft standardising the header across HTTP APIs
• RFC 9110 § 9.2.2 — Idempotent Methods — the normative HTTP specification definition
• MDN — Idempotent (glossary) — quick reference with per-method classification