Design Case Studies · Lesson 04

Design: Pub/Sub API

A publish/subscribe system lets producers fire events without knowing who's listening — but the moment you promise delivery guarantees, fan-out, and ordered processing at scale, every easy decision becomes a trade-off.

1 · Requirements

An engineering team comes to you with this brief: "We need a messaging layer so services can broadcast events to whoever cares, without the sender knowing the recipients." Before touching a single endpoint, translate that into concrete requirements.

Notice what is not a requirement: exactly-once delivery. That is extremely hard to guarantee at the infrastructure layer without sacrificing throughput. Practical systems instead guarantee at-least-once and ask consumers to be idempotent — the far cheaper trade-off.

2 · Design decisions

2a. Topic and subscription model

Think of a topic as a radio station — it just broadcasts. A subscription is a satellite dish pointed at that station. The broadcaster (producer) never knows how many dishes are listening, and each dish receives an independent copy of every broadcast.

2b. Delivery mode: webhook push vs streaming pull

How does the system get messages to subscribers? Two broad strategies:

This design supports both modes. When creating a subscription the caller chooses. See Debugging Webhooks (dbg-04) for the full webhook failure taxonomy.

2c. At-least-once delivery and idempotent consumers

Imagine you're mailing a parcel and the post office promises: "We'll keep re-sending until you confirm receipt." That's at-least-once delivery — the consumer might get the same parcel twice if the first acknowledgement was lost in transit. The parcel itself is fine; the consumer just has to be smart enough not to unbox it twice.

At-least-once is the practical default because it requires no distributed coordination. The alternative, exactly-once, demands two-phase commit semantics across producer, broker, and consumer — expensive latency overhead. See Idempotency (rel-02) for how to build idempotent consumers, and Event-driven & Pub/Sub (rel-10) for broker-level patterns.

2d. Per-key ordering

Consider a shopping cart: the sequence "add item → remove item → checkout" must be processed in that order. Per-key ordering (routing all events for the same cart ID to the same partition) achieves this while still allowing massive parallelism — all other cart IDs proceed independently. Global ordering forces a single partition and is almost never worth the throughput cost.

2e. Retries and dead-letter queue

When a subscriber's webhook returns a 5xx or times out, the message must be retried — but naively retrying immediately will hammer a struggling service. The right pattern is exponential back-off with jitter. After N failed attempts, the message is moved to a dead-letter queue (DLQ): a separate topic that operators can inspect, replay, or discard without losing the original message. See Event-driven & Pub/Sub (rel-10) for detailed DLQ design.

3 · The API model

Core endpoints

# 1. Create a topic
POST /v1/topics
# Request
{
  "name": "order.placed",
  "retention_hours": 168,
  "ordering_key": "orderId"
}
# Response 201
{
  "id": "top_8xKq",
  "name": "order.placed",
  "created_at": "2026-06-20T09:00:00Z"
}

# 2. Publish a message
POST /v1/topics/top_8xKq/messages
# Request
{
  "data": {
    "orderId": "ord_123",
    "total": 49.95,
    "currency": "USD"
  },
  "ordering_key": "ord_123",
  "idempotency_key": "pub_abc_20260620T090000"
}
# Response 202 — accepted for fan-out
{
  "message_id": "msg_7tNv",
  "status": "accepted"
}

# 3. Create a webhook subscription
POST /v1/subscriptions
# Request
{
  "topic_id": "top_8xKq",
  "name": "billing-handler",
  "delivery": {
    "type": "webhook",
    "url": "https://billing.internal/hooks/orders",
    "signing_secret": "whsec_..."
  },
  "retry_policy": {
    "max_attempts": 5,
    "backoff": "exponential",
    "initial_interval_ms": 1000
  },
  "dead_letter_topic_id": "top_dlq_billing"
}
# Response 201
{
  "id": "sub_4pLm",
  "status": "active"
}

# 4. Signed webhook delivery (system → subscriber)
POST https://billing.internal/hooks/orders  # called by the pub/sub system
X-PubSub-Signature: sha256=<hmac>
X-PubSub-Message-Id: msg_7tNv
X-PubSub-Attempt: 1
Content-Type: application/json

{
  "topic": "order.placed",
  "message_id": "msg_7tNv",
  "ordering_key": "ord_123",
  "published_at": "2026-06-20T09:00:01Z",
  "data": { "orderId": "ord_123", "total": 49.95 }
}

# Subscriber must respond 200/204 to acknowledge.
# Any other response or timeout triggers retry with back-off.

# 5. Streaming pull subscription (SSE)
GET /v1/subscriptions/sub_4pLm/messages?max_messages=10
Accept: text/event-stream

# Server streams events:
data: {"message_id":"msg_7tNv","data":{...},"ack_token":"ack_zZp9"}

# Consumer must acknowledge:
POST /v1/subscriptions/sub_4pLm/acknowledge
{ "ack_tokens": ["ack_zZp9"] }
# → 204 No Content

4 · Evaluation & latency budget

Delivery guarantees

At-least-once is the correct default. The table below shows why exactly-once is expensive:

Consumer idempotency

The message_id field is the natural idempotency key. Consumers should record "I have processed msg_7tNv" in their own datastore (a deduplicated events table works well) and skip re-processing. See Idempotency (rel-02) for the full implementation pattern.

Webhook latency budget

POST /v1/subscriptions/sub_4pLm/dlq/replay
{
  "from": "2026-06-01T00:00:00Z",
  "to":   "2026-06-20T23:59:59Z",
  "destination_subscription_id": "sub_4pLm"  # replay to same or different sub
}
# Response 202 — async, potentially thousands of messages
{
  "replay_id": "rpl_9Xk2",
  "message_count": 47,
  "status": "in_progress"
}

Under the hood: the core mechanism

A pub/sub broker is, at its core, a partitioned append-only log. Understanding what this means at the data structure level explains every delivery guarantee and ordering property in the system.

The topic as a partitioned log

A topic is divided into one or more partitions. Each partition is a durable append-only sequence of messages numbered from 0. Publishing a message appends it to the tail of one partition — the one selected by hashing the message's ordering_key. The sequence number within a partition is called the offset.

Partition assignment by ordering key

When a producer publishes with "ordering_key": "ord_123", the broker computes partition = hash(ordering_key) mod num_partitions. All messages for ord_123 land in the same partition, so they are appended and delivered in order. Messages for ord_456 may land in a different partition and proceed independently — that is how per-key ordering coexists with parallel throughput.

Worked offset / redelivery trace

Follow one message through the complete lifecycle — publish, fan-out, ack, and at-least-once redelivery:

# T=0: Producer publishes
POST /v1/topics/top_8xKq/messages
{
  "data": { "orderId": "ord_123" },
  "ordering_key": "ord_123"
}
broker receives → hash("ord_123") % 4 = partition_2
append to partition_2: offset 7   # now the durable tail
→ 202 { "message_id": "msg_7tNv" }

# T=1ms: Broker fans out to each subscription's cursor queue
# Sub A (billing): P2 offset was 7 → enqueue delivery job for offset 7
# Sub B (analytics): P2 offset was 4 → Sub B is behind; msg_7tNv will be
#                    delivered when Sub B reaches offset 7

# T=10ms: Worker delivers to Sub A via webhook
POST https://billing.internal/hooks/orders
X-PubSub-Message-Id: msg_7tNv
{ "data": { "orderId": "ord_123" } }

# Case A — subscriber returns 200 OK:
broker receives ack → advance Sub A cursor: P2 offset 7 → 8
# msg_7tNv is never redelivered to Sub A

# Case B — subscriber returns 500 / times out:
broker does NOT advance cursor (offset stays at 7)
retry attempt 2 after 1 s  (initial_interval_ms=1000)
retry attempt 3 after 2 s  (exponential back-off, base=2)
retry attempt 4 after 4 s
retry attempt 5 after 8 s
# attempt 5 still fails → message_id msg_7tNv moved to DLQ topic
# Sub A cursor advances past offset 7 (message quarantined, not lost)
# Sub A continues delivering offset 8, 9, … unblocked

# Sub B is independent: still at offset 4, will eventually reach 7
# and make its own delivery attempt for msg_7tNv

The key insight: the broker never deletes the message until the retention window expires. An offset is just a pointer. Redelivery means re-reading the same log position. The subscriber's acknowledgement is what advances the pointer — not the delivery itself.

Operating & debugging it

Most pub/sub problems are visible in two signals: subscription lag (the gap between the latest offset and the subscription's committed offset) and DLQ depth (messages that exhausted all retries). Both should be monitored as gauges with alerts.

Inspect a subscription's health

Symptom → cause → fix

1. Check subscription lag per partition — a lag spike on a single partition points to an ordering-key group, not the whole subscription.
2. Pull the most recent entry from the DLQ; read last_error — connection refused vs 5xx vs 200-but-no-ack are three different problems.
3. Verify the webhook endpoint manually: curl -X POST <webhook_url> with a sample payload and the signing header.
4. Check that your consumer code commits the ack after processing completes, not before.
5. After fixing the subscriber, use the DLQ replay endpoint to reprocess quarantined messages — preserve the original message_id for idempotent consumers.

Sources & further reading

• Google Cloud Pub/Sub — Architecture overview
• Amazon SNS — Simple Notification Service (fan-out patterns)
• Stripe — Webhook best practices (signed delivery, retries)
• Enterprise Integration Patterns — Dead Letter Channel
• Apache Kafka Documentation — log compaction, partitions, consumer groups