Platform & API Product Engineering · Lesson 04

Optimistic concurrency & conditional requests

Two API clients read the same resource, each modifies it, and the second write silently erases the first. This is the lost-update problem — one of the most insidious data-corruption bugs in distributed systems — and HTTP has a built-in, elegant mechanism to prevent it.

The lost-update problem

The lost-update problem occurs whenever two actors perform a read-modify-write cycle on the same resource and the operations interleave. Neither actor acts incorrectly on their own — the bug is in the interleaving. And it is silent: the second writer succeeds, returns 200, and the first writer's changes are gone with no error, no warning, and no trace in the response.

Think of two people editing the same Google Doc without the real-time sync. Person A saves a version with their changes; Person B opened the same original version, made different changes, and saves it five seconds later. Person A's changes vanish. Unlike Google Docs, most REST APIs have no live collaboration layer — the conflict is never visible unless the API is explicitly designed to surface it.

Optimistic concurrency control (OCC)

The fix is to carry the version of the resource through the write. A client that read version N must declare "I am updating version N" when it writes. The server performs a compare-and-swap: if the current version is still N, the write succeeds and the server increments to N+1. If the current version is now N+1 or higher (because someone else wrote in between), the server rejects the write with a 412 Precondition Failed.

The word "optimistic" refers to the strategy: you optimistically assume that no conflict will happen when you start the operation. You don't lock anything at read time. You only check for a conflict at write time, and if one occurred, you handle it then. This contrasts with pessimistic locking, which acquires an exclusive lock at read time and holds it until the write completes.

ETags: the HTTP mechanism

HTTP has a native primitive for this: the ETag (Entity Tag). An ETag is an opaque string the server includes in every response for a resource. It represents the current version of that resource. The client stores the ETag from the GET response and sends it back in the If-Match header on the PUT or PATCH. The server compares the incoming ETag against the current version. If they match, the resource has not changed since the client read it — proceed. If they don't match, someone else changed it — reject with 412.

# Step 1: Read the resource; server returns ETag
GET /orders/42 HTTP/1.1

# Response includes the version identifier
HTTP/1.1 200 OK
ETag: "v3"
Content-Type: application/json

{ "id": "42", "quantity": 5, "status": "pending" }

---
# Step 2: Write with the ETag in If-Match
PUT /orders/42 HTTP/1.1
If-Match: "v3"
Content-Type: application/json

{ "quantity": 10 }

# If no concurrent write: resource at version 3 → proceed
HTTP/1.1 200 OK
ETag: "v4"
{ "id": "42", "quantity": 10 }

# If concurrent write happened: resource now at version 4 → reject
HTTP/1.1 412 Precondition Failed
{ "error": { "type": "precondition_failed", "code": "version_conflict",
             "message": "The resource was modified since you last read it." } }

ETag variants: version number vs. content hash; weak vs. strong

An ETag can be computed two ways, and the difference matters for behaviour.

Version number vs. content hash

Weak vs. strong ETags

A strong ETag (the default, no prefix) means byte-for-byte identity. If two representations have the same strong ETag, every byte of their body is identical. The HTTP spec requires strong ETags for range requests and for If-Match conditional writes. A weak ETag has a W/ prefix: W/"v3". It means "semantically equivalent" — the logical content is the same even if the serialisation differs (different whitespace, field order). Weak ETags can only be used with If-None-Match for cache validation, not with If-Match for conditional writes. Always use strong ETags for concurrency control.

# Strong ETag (byte-identical — required for If-Match)
ETag: "a3f4b291c8e2"

# Weak ETag (semantically equivalent — NOT valid for If-Match)
ETag: W/"a3f4b291c8e2"

The conditional request header family

ETags are consumed by a family of conditional HTTP headers. Each header expresses a precondition that must be satisfied for the server to process the request.

If-None-Match: * for conditional CREATE

The If-None-Match: * header turns a PUT into a conditional create: "create this resource only if it does not already exist." This is valuable for safe idempotency — a client can retry a create without worrying about creating duplicates:

# Conditional create — only succeeds if /users/ada doesn't exist yet
PUT /users/ada HTTP/1.1
If-None-Match: *
Content-Type: application/json

{ "name": "Ada Lovelace", "email": "ada@math.io" }

# If /users/ada does NOT exist: 201 Created
# If /users/ada already EXISTS: 412 Precondition Failed

409 Conflict vs. 412 Precondition Failed

Both 409 and 412 signal that a write was rejected due to state, but they mean different things and the distinction matters for client handling.

# Retry-on-conflict loop — pseudo-code
function update_with_occ(resource_url, change_fn, max_retries=5):
  for attempt in range(max_retries):
    # Step 1: Read and capture the ETag
    response = GET(resource_url)
    etag     = response.headers['ETag']
    current  = response.body

    # Step 2: Compute the desired new state
    desired = change_fn(current)  # apply your business logic to the CURRENT read

    # Step 3: Attempt the conditional write
    write_response = PUT(resource_url, body=desired,
                         headers={'If-Match': etag})

    if write_response.status == 200:
      return write_response.body  # success

    if write_response.status == 412:
      continue  # conflict: loop will re-read with new ETag

    raise UnexpectedError(write_response.status)

  raise ConflictExhausted("Could not commit after " + max_retries + " attempts")

OCC vs. pessimistic locking vs. last-write-wins vs. CRDTs

Under the hood: the compare-and-swap mechanism

The ETag check at the database layer is a compare-and-swap (CAS). Understanding how this works prevents a critical implementation mistake: doing the compare-then-write as two separate database operations.

-- WRONG: non-atomic check + update (race condition possible)
-- Between SELECT and UPDATE, another request might change the row
SELECT version FROM orders WHERE id = 42;
-- application checks: is version == 3?
UPDATE orders SET quantity = 10, version = 4 WHERE id = 42;

---

-- CORRECT: atomic compare-and-swap in a single UPDATE
-- The WHERE clause includes the version check: if the row is no longer at
-- version=3, zero rows are affected, and we know there was a conflict.
UPDATE orders
SET    quantity = 10, version = version + 1
WHERE  id = 42
  AND  version = 3;  -- the atomic compare is here, in the WHERE clause

-- rows_affected = 1 → success → return 200 + new ETag
-- rows_affected = 0 → conflict → return 412

This single-statement pattern is safe even under concurrent writes because the database engine guarantees row-level locking for the duration of the UPDATE statement. Two simultaneous updates on the same row will serialise at the database level. One will find version = 3 and succeed (rows_affected = 1). The other will find version = 4 (already incremented) and fail (rows_affected = 0). No explicit transaction needed; no window between check and write.

By the numbers

How likely is a conflict, and how many retries should you expect? The math depends on the write rate for a given resource and the transaction window — the time between a client reading the resource and submitting the write.

Let r = write rate on a resource (writes per second), w = transaction window (seconds, i.e. the time the client takes to read, compute, and write). The probability that at least one other write lands during the client's window is approximately:

P(conflict) ≈ 1 − e^(−r·w)   [Poisson arrival model]

For small r·w:  P(conflict) ≈ r·w   [first-order approximation]

Expected retries = P(conflict) / (1 − P(conflict))
                 ≈ r·w           [for small P(conflict)]

Worked trace — a popular order resource receiving r = 2 writes/second, client transaction window w = 0.5 s (500 ms round trip + compute):

The break-even where OCC becomes more expensive than pessimistic locking (due to retry amplification) is roughly at P(conflict) > 20–30%. Below that, OCC's zero-lock-overhead advantage dominates. Above that, consider sharding the hot resource (splitting into per-worker state), or switching to a conflict-free model (CRDT, queue-based serial processing).

The formula also quantifies the benefit of reducing the transaction window. Cutting w from 500 ms to 100 ms at r = 2/s drops P(conflict) from 63% to 18% — a dramatic improvement achievable just by caching the local compute or moving the client closer to the server.

How real platforms do it

How to debug & inspect it

OCC bugs fall into two categories: false conflicts (412 when no real conflict exists) and missed conflicts (concurrent writes both succeed when they should have conflicted). Both are diagnosable with curl and careful log inspection.

Trace ETags with curl

Symptom → cause → fix

Sources & further reading

• RFC 9110 §13 — Conditional Requests (ETags, If-Match, If-None-Match)
• AWS S3 — PutObject conditional headers (If-Match, If-None-Match)
• Google Cloud Storage — Request preconditions
• etcd v3 API — Transactions (compare-and-swap)
• Stripe PaymentIntent lifecycle
• PostgreSQL — Transaction isolation
• Lesson as-02 — REST design
• Reliability 02 — Idempotency