Platform & API Product Engineering · Lesson 02

Designing a Webhook Delivery System

A webhook is not just "POST to a URL when something happens." It is a delivery subsystem with fan-out, per-endpoint queuing, HMAC-signed payloads, exponential-backoff retries, dead-letter queues, circuit-breaking for failing endpoints, and a replay API. Each of those pieces exists to solve a specific failure mode — skip one and your platform will eventually wake engineers at 3 AM.

The problem a webhook system must solve

When your platform has ten thousand apps each subscribed to order.created events, and you receive 1,000 orders per second, your delivery system must POST to ten thousand distinct URLs at a sustained rate without any single slow or unresponsive endpoint holding up the other 9,999. That sentence describes the hard part: isolation. If your system processes deliveries with a single shared queue and worker pool, one endpoint that takes 30 seconds to respond will occupy a worker thread for 30 seconds, starving deliveries to healthy endpoints.

Think of it like postal sorting. A single queue where every package, from every sender to every recipient, must pass through one conveyor belt means a stuck package blocks the entire line. Real postal systems sort by destination and run separate belts per route. Webhooks need the same: a separate queue per endpoint, so that a dead endpoint accumulates its undeliverable backlog without touching anyone else's.

There is a second hard problem: at-least-once delivery with idempotent consumers. Your delivery system will retry failed deliveries. Retries mean duplicate deliveries are possible. Consumers must handle receiving the same event payload twice without double-processing — an order event delivered twice should not charge a customer twice.

The full delivery pipeline

Step 1 — Event production and durable storage

Every event must be written to durable storage before anything downstream happens. If you enqueue directly to an in-memory queue and the process crashes, the event is lost. The correct order is: write the event to a durable store (a database table or an event log like Kafka), then return 200 to the caller who triggered it, then asynchronously fan-out to subscriptions.

# Event creation — pseudo-code for the API server that produces events
function create_order(customer_id, items):
  order = DB.insert_order(customer_id, items)        # primary write

  event = {
    "id":         generate_uuid(),                    # stable deduplication key
    "type":       "order.created",
    "api_version": "2024-07-01",
    "created_at": now_iso8601(),
    "data":       { "order_id": order.id, ... }
  }

  DB.insert_event(event)                             # durable event record
  QUEUE.push("events:fan_out", event.id)             # fan-out job (by id, not full payload)
  return order

Notice the fan-out queue stores the event ID, not the full payload. When the fan-out worker runs, it reads the event from the durable store. This means requeuing (on failure) does not create duplicate records, and the fan-out worker always reads the canonical event state.

Step 2 — Subscription matching and fan-out

The fan-out worker reads each event ID, fetches the event from the store, then queries for all active subscriptions matching that event type. For each matching subscription it writes one delivery job to that endpoint's dedicated queue.

# Fan-out worker
function fan_out_event(event_id):
  event         = DB.get_event(event_id)
  subscriptions = DB.find_subscriptions(
                    event_type=event.type,
                    status='active'
                  )                                  # index on (event_type, status)

  for sub in subscriptions:
    delivery = DB.insert_delivery({
      "id":          generate_uuid(),
      "event_id":    event.id,
      "endpoint_id": sub.endpoint_id,
      "attempt":     1,
      "status":      "pending",
      "next_attempt_at": now()
    })
    QUEUE.push("webhook:deliver:" + sub.endpoint_id, delivery.id)

The key naming convention — webhook:deliver:{endpoint_id} — means each endpoint gets its own queue. A worker process consumes from one endpoint's queue at a time, so a slow endpoint only blocks that endpoint's worker, not the global pool.

Per-endpoint isolation and circuit-breaking

This is the most important architectural property of a robust webhook system. One slow or unreachable endpoint must not stall deliveries to healthy endpoints. Per-endpoint queues provide isolation; circuit-breaking provides relief for both the platform and the endpoint owner.

Circuit-breaker rules for endpoints

A circuit breaker for an endpoint is simpler than a service mesh circuit breaker — you are tracking one consumer's reliability over time, not a backend service. A practical implementation:

# Circuit breaker state — per endpoint
endpoint_state = {
  "status":            "closed",  # closed = active, open = disabled
  "consecutive_failures": 0,
  "opened_at":         null,
  "last_attempt_at":   null,
  "success_rate_24h":  1.0
}

# After each delivery attempt:
if delivery succeeded (2xx):
  consecutive_failures = 0
  if status == "half-open":
    status = "closed"           # probe succeeded → reopen

elif delivery failed (non-2xx or timeout):
  consecutive_failures += 1
  if consecutive_failures >= 5:
    status = "open"
    opened_at = now()
    NOTIFY_OWNER(endpoint_id)  # email / dashboard alert

# Probing: after circuit has been open for T minutes, try one request
if status == "open" AND now() - opened_at > PROBE_INTERVAL:
  status = "half-open"           # allow one probe attempt

Step 3 — HTTP delivery with HMAC signatures

Every webhook POST must be signed so the receiver can verify the payload came from your platform and has not been tampered with in transit. The standard mechanism is HMAC-SHA256 over the timestamp and raw request body, with the signature included in the request headers.

The signing algorithm in full

# Sender: compute the signature before POSTing
function sign_payload(secret, body):
  t               = str(current_unix_timestamp())
  signing_input   = t + "." + body              # byte-level concat of timestamp + "." + raw JSON
  sig             = hmac_sha256(key=secret, msg=signing_input)
  sig_hex         = hex(sig)
  return {
    "Webhook-Timestamp": t,
    "Webhook-Signature": "sha256=" + sig_hex
  }

# Receiver: verify before processing
function verify_signature(secret, timestamp, body, provided_sig):
  # 1. Replay defense: reject if timestamp is too old or in the future
  age = abs(current_unix_timestamp() - int(timestamp))
  if age > 300:                                # 5-minute tolerance window
    return REJECT, "timestamp too old or too far in future"

  # 2. Recompute expected signature using same input as sender
  signing_input   = timestamp + "." + body
  expected_sig    = "sha256=" + hex(hmac_sha256(key=secret, msg=signing_input))

  # 3. Constant-time compare — never use == for signatures (timing oracle)
  return hmac_compare_digest(expected_sig, provided_sig)

Secret rotation: multiple active secrets

Webhook secrets need to be rotatable without breaking in-flight deliveries. The pattern is to allow multiple active secrets per endpoint simultaneously for a short overlap window:

1. The endpoint owner generates a new secret in the dashboard. Both old and new secrets are now active for this endpoint.
2. The delivery worker signs with the newest secret but includes the key ID in the header (e.g. Webhook-Signature: sha256={sig},keyId={kid}).
3. The receiver verifies against all currently active secrets for the endpoint. If any matches, the request is accepted.
4. After a configurable grace period (e.g., 24 hours), the old secret is retired. The receiver should by then have deployed code using the new secret.

Step 4 — Retry strategy with exponential backoff

Any non-2xx response (4xx excluding 429, 5xx, or a connection timeout) is a delivery failure. The delivery must be rescheduled with an exponential delay so that a struggling endpoint is not hammered into the ground while it recovers.

# Retry schedule — exponential backoff with jitter, per delivery attempt
function next_attempt_delay(attempt, base=5, cap=3600):
  delay_s = min(cap, base * (2 ** attempt))     # exponential: 5, 10, 20, 40, 80, 160, 320, 640, 1280, 2560, 3600, 3600...
  jitter  = random(0, delay_s * 0.2)           # ±20% jitter to desynchronize retries
  return delay_s + jitter

# On delivery failure:
if delivery.attempt <= MAX_ATTEMPTS:             # MAX_ATTEMPTS = typically 10–17
  delivery.status      = "retrying"
  delivery.next_attempt = now() + next_attempt_delay(delivery.attempt)
  QUEUE.schedule_at(delivery.id, delivery.next_attempt)
else:
  delivery.status = "failed"
  DLQ.push(delivery.id)                          # dead-letter queue
  UPDATE_ENDPOINT_FAILURE_STATS(delivery.endpoint_id)

With base=5, cap=3600, the 10 retry schedule (modeled) is approximately:

Stripe retries over approximately 3 days with increasing delays. GitHub retries with increasing delays over 72 hours. The specific schedule is a product decision — short windows mean customers find out about problems quickly; long windows tolerate infrastructure hiccups but delay failure notification.

Step 5 — Dead-letter queue and replay API

After the final retry, deliveries that could not be confirmed successful move to a dead-letter queue (DLQ). The DLQ is not a dustbin — it is a durable log of deliveries that require human attention. The replay API allows endpoint owners to inspect and re-trigger deliveries from the DLQ after they have fixed whatever caused the failures.

# Replay API — allows endpoint owner to re-enqueue failed deliveries
# POST /v1/webhooks/endpoints/{endpoint_id}/deliveries/{delivery_id}/replay

function replay_delivery(endpoint_id, delivery_id, caller_auth):
  delivery = DB.get_delivery(delivery_id)
  assert delivery.endpoint_id == endpoint_id        # ownership check
  assert caller_auth.has_permission("webhooks:write")

  # Create a new delivery attempt linked to the original event
  new_delivery = DB.insert_delivery({
    "event_id":    delivery.event_id,               # same event, new attempt
    "endpoint_id": delivery.endpoint_id,
    "attempt":     1,                               # reset attempt counter
    "status":      "pending",
    "replayed_from": delivery.id
  })
  QUEUE.push("webhook:deliver:" + delivery.endpoint_id, new_delivery.id)
  return new_delivery

Under the hood: idempotency and duplicate handling

At-least-once delivery guarantees that some successful deliveries will result in the customer receiving the same event twice — this is unavoidable without distributed transactions between your delivery confirmation and the customer's processing confirmation. The receiver must handle duplicates without re-processing them.

The mechanism: every event carries a stable id (a UUID). The consumer's handler checks whether it has already processed this ID before acting:

# Consumer: idempotent event handler
function handle_order_created(payload):
  event_id = payload["id"]

  # Attempt insert into a processed-events table with a UNIQUE constraint on event_id
  try:
    DB.insert_idempotency_key(event_id, processed_at=now())
  except UniqueViolation:
    return HTTP_200  # already processed — acknowledge without re-running

  # Safe to process — won't run twice for the same event_id
  create_order_in_crm(payload["data"])
  return HTTP_200

By the numbers

Delivery throughput and worker sizing

Scenario: 1,000,000 events/day across 20 subscribed apps = up to 20,000,000 deliveries/day. Average HTTP delivery latency to a healthy endpoint: 150 ms.

delivery_rate  = 20,000,000 / 86,400 ≈ 231 deliveries/s   (modeled)

# Little's Law: L = λ × W
# L = average in-flight deliveries (concurrency needed)
# λ = arrival rate (deliveries/s), W = average service time (s)

concurrency    = 231 deliveries/s × 0.150 s = 34.7 → 35 concurrent workers  (modeled)

Each worker handles one delivery at a time (one open HTTP connection). You need approximately 35 worker threads to sustain the base delivery rate at 150 ms average latency. At a 500 ms average (slower endpoints), you need 231 × 0.5 = 116 workers. This is why per-endpoint concurrency caps matter — a single endpoint with 30-second latency would tie up 6,930 workers if uncapped.

Retry amplification

When 10% of endpoints are unhealthy (a realistic scenario during a downstream incident), retry load amplifies the delivery volume:

base_deliveries        = 231/s
unhealthy_endpoints    = 10%
retried_deliveries     = 231 × 0.10 × (MAX_ATTEMPTS - 1)
                       = 23.1 × 16 retries  ≈  370/s additional  (modeled)

total_peak_delivery_rate = 231 + 370 ≈ 600/s   (2.6× amplification factor)
workers_needed_at_peak   = 600 × 0.150  ≈  90 workers

This is the retry storm: an incident affecting 10% of endpoints more than doubles the delivery load on your infrastructure. Mitigations: (1) per-endpoint concurrency caps to prevent one bad endpoint consuming disproportionate workers; (2) circuit-breaking to stop retrying open-circuit endpoints; (3) exponential backoff to spread retries over time rather than concentrating them.

Queue depth as a health signal

# Alert thresholds — model these for your own SLA
warn  when queue_depth > 100     AND p99_delivery_age > 30 s    (modeled)
page  when queue_depth > 1000    OR  p99_delivery_age > 300 s   (modeled)

# Per-endpoint success rate over rolling 1 hour:
success_rate = successful_deliveries / total_attempts
open_circuit when success_rate < 0.50 over last 20 attempts      (modeled)

Trade-offs: design decisions you will face

How real platforms do it

Sources & further reading

• Stripe — Webhooks documentation (signature verification, retry schedule, replay)
• Stripe — Stripe-Signature header and verification algorithm
• GitHub — Validating webhook deliveries (X-Hub-Signature-256)
• GitHub — Webhook redelivery API
• Shopify — Verifying webhook signatures
• Svix — How webhook signature verification works
• RFC 2104 — HMAC: Keyed-Hashing for Message Authentication
• rel-12 — Webhooks intro
• dbg-04 — Debugging webhooks
• rel-10 — Event-driven pub-sub
• sim-04 — Queue backpressure simulator