API Design

Design Case Studies · Lesson 21

Design: E-commerce Saga Checkout API

In a microservice-based e-commerce platform, checking out requires updating multiple independent services (order, payment, inventory). Because these services run on separate databases, we cannot use a single SQL database transaction. Two-Phase Commit (2PC) blocks resources and fails to scale. We must design a distributed transaction workflow using the Saga Pattern, compensating transactions, and the Transactional Outbox pattern.

⏱ ~17 min Advanced Prereq: cs-12, rel-02, webhooks

By the end you'll be able to

Requirements

Designing a checkout flow across microservices requires strict reliability and consistency guarantees:

Design decisions

Saga Orchestration vs. Choreography

A Saga is a sequence of local transactions. Each transaction updates data within a single service and publishes a message to trigger the next step. If a step fails, the Saga runs compensating transactions to undo the changes.

Metric Saga Choreography (Event-Driven) Saga Orchestration (Centralized)
How it works Services publish events to a shared queue; other services listen and react autonomously. A dedicated **Orchestrator Service** instructs participating services what to do and when.
Coupling Low. Services do not know about each other; they only bind to shared event topics. High. The orchestrator must understand the API endpoints and payloads of all services.
Complexity High. Hard to trace which service failed or construct a global state view. Low. The orchestrator maintains the state machine, logs progress, and manages failures.
Best for Simple workflows with 2-3 steps and high event-driven throughput. Complex workflows (e.g. checkout, hotel+flight booking) requiring step-by-step auditing.

We choose **Saga Orchestration** for checkout. It keeps the core business logic (order of operations, rollback rules) visible in a single state machine rather than distributed across multiple event consumer handlers.

Transactional Outbox Pattern

When a service completes a transaction (e.g., creating an order), it must publish an event to the orchestrator. If the database write succeeds but the server crashes before sending the event, the system drifts. If it sends the event first but the DB write fails, it is even worse. We solve this with the **Transactional Outbox Pattern**: the application writes both the business record (the order) and an outbox record (the event message) to the same database using a local database transaction. A separate publisher process reads the outbox table and pushes the messages to the broker, guaranteeing **at-least-once** delivery.

The API model

Checkout API (Orchestrator Entry)

The client submits the checkout request. The request payload contains an idempotency_key to prevent double charges.

# Client initiates checkout
POST /v1/checkout HTTP/1.1
Host: checkout-orchestrator.example.com
Content-Type: application/json
Idempotency-Key: idemp_chk_99210aa

{
  "user_id": "usr_alice",
  "cart_id": "cart_772",
  "payment_method_id": "pm_card_visa_882"
}

# Orchestrator creates saga session and returns immediately
HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "saga_id": "saga_chk_abc123",
  "status": "processing"
}

Inventory Service: Hold and Release (Compensating API)

Sagas require all non-read steps to be **reversible**. The inventory service must expose a reserve (hold) endpoint and a compensating release endpoint.

# 1. Hold inventory items (Normal Step)
POST /v1/inventory/hold HTTP/1.1
Idempotency-Key: idemp_saga_abc123_inv

{
  "saga_id": "saga_chk_abc123",
  "items": [
    { "item_id": "item_book_1", "qty": 1 }
  ]
}

# 2. Release inventory items (Compensating Step, run if payment fails)
POST /v1/inventory/release HTTP/1.1
Idempotency-Key: idemp_saga_abc123_release

{
  "saga_id": "saga_chk_abc123"
}

Under the hood: Orchestrated checkout flow

The checkout orchestrator drives the state machine, communicating with services sequentially and rolling back on failure.

Client Orchestrator Inventory Payment 1. POST /checkout 2. Hold Inventory Inventory Reserved (Success) 3. Charge Payment Payment Declined (FAIL) 4. Rollback: Release Inventory Inventory Released (Ack) 5. Checkout Failed 200 OK (status=failed)

By the numbers: saga retry budgets & queue latency

Let's evaluate the performance overhead and recovery curves when distributed transaction steps fail.

Governing Equations

Scenario Parameters

Worked Calculations: Happy Path vs. Payment Failure Latency

Execution Path Steps Executed Calculated Latency
Happy Path (Success) Order creation + Inventory hold + Payment charge $(15+8) + (15+5) + (15+240) = \mathbf{298\text{ ms}}$
Step 2 Failure (Inventory unavailable) Order creation + Inventory hold (fails) + Order cancellation (compensate) $(15+8) + (15+3\text{ fail}) + (15+4\text{ compensate}) = \mathbf{60\text{ ms}}$
Step 3 Failure (Payment declined) Happy path steps (1 & 2) + Payment (declines) + Inventory release (comp) + Order cancellation (comp) $298 + (15+6\text{ comp2}) + (15+4\text{ comp1}) = \mathbf{338\text{ ms}}$
🧮 Step-by-step arithmetic: calculating the outbox queue backup rate

Suppose your checkout service handles a surge of 200 checkouts/second. Your outbox publisher process polls the database every 100ms. What happens if the outbox publisher encounters network lag that limits its message publishing rate to 150 messages/second?

  1. Compute backlog accumulation rate: $$Backlog_{rate} = 200 \text{ checkouts/s} - 150 \text{ published/s} = 50 \text{ messages/s}$$
  2. Compute backlog after a 5-minute (300s) network degradation: $$Backlog_{total} = 50 \text{ messages/s} \times 300 \text{ seconds} = 15,000 \text{ events}$$
  3. Compute catch-up time once network recovery allows publishing at 300 messages/second: Once recovery occurs, the net publishing speed is $300 - 200 \text{ incoming} = 100 \text{ messages/s}$. $$T_{catch\_up} = \frac{15,000 \text{ events}}{100 \text{ events/s}} = 150 \text{ seconds (2.5 minutes)}$$

This illustrates why outbox queues must use high-throughput database partitioning and why you need monitoring alerts for outbox queue lag.

How to debug & inspect it

To inspect and audit distributed Sagas, trace the workflow identifiers across systems and verify that the outbox table events are being drained properly.

# 1. Query the Outbox table to locate stuck/unpublished messages $ pgcli -d order_db -c "SELECT id, event_type, payload, status FROM outbox WHERE status = 'PENDING';" +----+------------+---------------------------------------------+---------+ | id | event_type | payload | status | +----+------------+---------------------------------------------+---------+ | 45 | ORDER_CRE | {"saga_id": "saga_123", "user_id": "usr_9"} | PENDING | | 46 | ORDER_CRE | {"saga_id": "saga_124", "user_id": "usr_2"} | PENDING | +----+------------+---------------------------------------------+---------+ # Note: If messages stay in PENDING, your outbox publisher service is down! # 2. Trace distributed logs across microservices using OpenTelemetry trace IDs $ grep "trace_id=ot_chk_882910" /var/log/inventory.log [INFO] trace_id=ot_chk_882910 saga_id=saga_123 Reserved 1 qty of item_id=book_1 $ grep "trace_id=ot_chk_882910" /var/log/payment.log [ERROR] trace_id=ot_chk_882910 saga_id=saga_123 Stripe card charge declined (Insufficient Funds) $ grep "trace_id=ot_chk_882910" /var/log/inventory.log [INFO] trace_id=ot_chk_882910 saga_id=saga_123 COMPENSATE: Released 1 qty of item_id=book_1

Use the guide below to resolve common failures in Saga-based checkout architectures:

Symptom Likely Cause Fix
Duplicate orders created when clients retry checkout due to network timeouts Lack of idempotency check at the Orchestrator or Order Service layer Enforce a unique constraint on idempotency_key in the database; return cached response on match.
"Ghost Inventory Lock": items remain locked in the warehouse, but no order was completed A compensating transaction (release) failed to execute or crashed mid-rollback Implement dead-letter queues (DLQ) for failed rollbacks; orchestrator must retry compensating steps infinitely.
The transactional outbox table grows indefinitely, causing slow DB queries Drained outbox rows are not being deleted or archived Configure the outbox publisher to prune successfully published rows immediately, or partition the table.

🧠 Quick check

1. Why is Two-Phase Commit (2PC) avoided for high-throughput microservice checkouts?

Two-Phase Commit relies on a coordinator holding global locks on database tables. If one service experiences network latency, all other services' resources remain locked, dragging down system availability.

2. What is the defining characteristic of a "compensating transaction" in a Saga?

A compensating transaction cannot simply perform a database "rollback" because the local transaction was already committed. Instead, it runs an explicit business undo operation (e.g. issuing a refund API call).

3. What failure mode does the Transactional Outbox pattern prevent?

By writing both the business record and the outbox event payload inside the same local database transaction, you guarantee that either both are saved or both are rejected, preventing data drift.

4. In Saga Orchestration, where does the execution state and logic live?

Saga Orchestration uses a centralized coordinator service (orchestrator) that acts as a state machine. It is responsible for calling API endpoints, tracking step verdicts, and executing rollbacks on failure.

✍️ Exercise: design the compensation outbox schema

A customer initiates checkout. The orchestrator reserves inventory (Step 1) and charges payment (Step 2). While trying to create the order in the Order Database (Step 3), the Order DB is down. The orchestrator must roll back.

Design the Outbox table schema for the Orchestrator Database itself to ensure that the compensating rollback (Release Inventory) is guaranteed to execute even if the orchestrator server crashes mid-rollback.


Model answer:

To guarantee that compensating transactions are never lost during orchestrator crashes, the orchestrator must persist the saga's state and pending rollback tasks in its local database before attempting any network calls.

Orchestrator Saga Job Table Schema (SQL):

CREATE TABLE saga_executions (
    saga_id VARCHAR(64) PRIMARY KEY,
    status VARCHAR(20) NOT NULL,          -- 'STARTED', 'COMPENSATING', 'FAILED', 'SUCCESS'
    current_step VARCHAR(50) NOT NULL,
    payload JSONB NOT NULL,               -- Holds checkout parameters
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
);

CREATE TABLE saga_compensating_tasks (
    task_id VARCHAR(64) PRIMARY KEY,
    saga_id VARCHAR(64) REFERENCES saga_executions(saga_id),
    target_service VARCHAR(50) NOT NULL,  -- 'INVENTORY_SERVICE', 'PAYMENT_SERVICE'
    payload JSONB NOT NULL,               -- Parameters for compensating API
    status VARCHAR(20) NOT NULL,          -- 'PENDING', 'SENT', 'COMPLETED'
    retry_count INT DEFAULT 0,
    next_retry_at TIMESTAMP NOT NULL
);

Rollback Execution Logic:

  1. When Step 3 fails, the orchestrator updates saga_executions.status = 'COMPENSATING' and inserts a task in saga_compensating_tasks with status = 'PENDING'. This happens inside a single SQL transaction on the orchestrator DB.
  2. An asynchronous background worker polls the saga_compensating_tasks table for PENDING rows, makes the REST call to Inventory Service /v1/inventory/release, and updates status to COMPLETED upon receipt of 200 OK.
  3. If the orchestrator crashes, the next spawned orchestrator node scans the task table, finds the unfinished task, and retries it. This ensures eventual consistency despite server crashes.

Key takeaways

  • **Avoid 2PC at scale**. Two-Phase Commit holds resources hostage. Use the Saga pattern for asynchronous eventual consistency across microservice databases.
  • **Orchestration vs. Choreography**. Choose Orchestration when workflows have many steps, loops, or complex rollback paths; it centralizes logging and visibility.
  • **All steps must be reversible**. Sagas rely on compensating transactions. For every endpoint that mutates state, design a corresponding endpoint to reverse it.
  • **Outbox prevents drift**. Use the Transactional Outbox pattern to write business records and outgoing messages atomically to the database, ensuring at-least-once delivery.
  • **Retries must be infinite for rollbacks**. A failed normal step triggers a rollback. Rollback actions must be retried until they succeed; otherwise, data drifts permanently.

Sources & further reading