Design Case Studies · Lesson 21
Design: E-commerce Saga Checkout API
In a microservice-based e-commerce platform, checking out requires updating multiple independent services (order, payment, inventory). Because these services run on separate databases, we cannot use a single SQL database transaction. Two-Phase Commit (2PC) blocks resources and fails to scale. We must design a distributed transaction workflow using the Saga Pattern, compensating transactions, and the Transactional Outbox pattern.
By the end you'll be able to
- Explain why Two-Phase Commit (2PC) fails to scale and choose between Saga Orchestration and Choreography.
- Design endpoints for compensating transactions that reverse intermediate steps upon workflow failures.
- Implement the Transactional Outbox pattern to prevent database-to-message-queue state drift.
- Compute retry budgets and recovery latency when rollbacks occur during checkout failures.
Requirements
Designing a checkout flow across microservices requires strict reliability and consistency guarantees:
- Atomicity. Either the order is placed, payment captured, and inventory deducted, OR everything is rolled back to a clean state. Partial states (payment captured but no order created) are unacceptable.
- Availability. The payment gateway or inventory database might be down. The API must handle partial failures gracefully, queueing retries or executing rollbacks.
- Idempotency. Network failures will cause clients to retry checkout requests. The API must ensure duplicate checkout requests never result in multiple charges or double-deducted inventory.
Design decisions
Saga Orchestration vs. Choreography
A Saga is a sequence of local transactions. Each transaction updates data within a single service and publishes a message to trigger the next step. If a step fails, the Saga runs compensating transactions to undo the changes.
| Metric | Saga Choreography (Event-Driven) | Saga Orchestration (Centralized) |
|---|---|---|
| How it works | Services publish events to a shared queue; other services listen and react autonomously. | A dedicated **Orchestrator Service** instructs participating services what to do and when. |
| Coupling | Low. Services do not know about each other; they only bind to shared event topics. | High. The orchestrator must understand the API endpoints and payloads of all services. |
| Complexity | High. Hard to trace which service failed or construct a global state view. | Low. The orchestrator maintains the state machine, logs progress, and manages failures. |
| Best for | Simple workflows with 2-3 steps and high event-driven throughput. | Complex workflows (e.g. checkout, hotel+flight booking) requiring step-by-step auditing. |
We choose **Saga Orchestration** for checkout. It keeps the core business logic (order of operations, rollback rules) visible in a single state machine rather than distributed across multiple event consumer handlers.
Transactional Outbox Pattern
When a service completes a transaction (e.g., creating an order), it must publish an event to the orchestrator. If the database write succeeds but the server crashes before sending the event, the system drifts. If it sends the event first but the DB write fails, it is even worse. We solve this with the **Transactional Outbox Pattern**: the application writes both the business record (the order) and an outbox record (the event message) to the same database using a local database transaction. A separate publisher process reads the outbox table and pushes the messages to the broker, guaranteeing **at-least-once** delivery.
The API model
Checkout API (Orchestrator Entry)
The client submits the checkout request. The request payload contains an idempotency_key to prevent double charges.
# Client initiates checkout
POST /v1/checkout HTTP/1.1
Host: checkout-orchestrator.example.com
Content-Type: application/json
Idempotency-Key: idemp_chk_99210aa
{
"user_id": "usr_alice",
"cart_id": "cart_772",
"payment_method_id": "pm_card_visa_882"
}
# Orchestrator creates saga session and returns immediately
HTTP/1.1 202 Accepted
Content-Type: application/json
{
"saga_id": "saga_chk_abc123",
"status": "processing"
}
Inventory Service: Hold and Release (Compensating API)
Sagas require all non-read steps to be **reversible**. The inventory service must expose a reserve (hold) endpoint and a compensating release endpoint.
# 1. Hold inventory items (Normal Step)
POST /v1/inventory/hold HTTP/1.1
Idempotency-Key: idemp_saga_abc123_inv
{
"saga_id": "saga_chk_abc123",
"items": [
{ "item_id": "item_book_1", "qty": 1 }
]
}
# 2. Release inventory items (Compensating Step, run if payment fails)
POST /v1/inventory/release HTTP/1.1
Idempotency-Key: idemp_saga_abc123_release
{
"saga_id": "saga_chk_abc123"
}
Under the hood: Orchestrated checkout flow
The checkout orchestrator drives the state machine, communicating with services sequentially and rolling back on failure.
By the numbers: saga retry budgets & queue latency
Let's evaluate the performance overhead and recovery curves when distributed transaction steps fail.
Governing Equations
- Outbox Polling Latency: If the outbox publisher queries the outbox table every $T_{poll}$ ms, and uses database polling: $$Latency_{outbox\_pub} \approx \frac{T_{poll}}{2} + T_{publish}$$ We can reduce this to near-zero ($<5$ms) by using CDC (Change Data Capture, e.g. Debezium) tailing the database WAL logs.
- Total Workflow Execution Time (Happy Path): $$T_{success} = \sum_{i=1}^{M} (R_{round\_trip\_i} + T_{process\_i})$$ Where $M$ is the number of services, $R$ is the network round-trip, and $T$ is the internal processing latency.
- Total Workflow Time under Failure & Rollback: If step $K$ fails, we run compensating transactions for all successful prior steps ($1$ to $K-1$): $$T_{failure} = \sum_{i=1}^{K} (R_i + T_i) + \sum_{j=1}^{K-1} (R_{compensate\_j} + T_{compensate\_j})$$
Scenario Parameters
- Network round-trip latency ($R$): 15 ms (between microservices)
- Normal Steps:
- Step 1 (Create Order DB): $T_1 = 8\text{ ms}$
- Step 2 (Reserve Inventory): $T_2 = 5\text{ ms}$
- Step 3 (Charge Stripe Card): $T_3 = 240\text{ ms}$ (out-of-network call)
- Compensating Steps:
- Release Inventory: $T_{comp2} = 6\text{ ms}$
- Cancel Order: $T_{comp1} = 4\text{ ms}$
Worked Calculations: Happy Path vs. Payment Failure Latency
| Execution Path | Steps Executed | Calculated Latency |
|---|---|---|
| Happy Path (Success) | Order creation + Inventory hold + Payment charge | $(15+8) + (15+5) + (15+240) = \mathbf{298\text{ ms}}$ |
| Step 2 Failure (Inventory unavailable) | Order creation + Inventory hold (fails) + Order cancellation (compensate) | $(15+8) + (15+3\text{ fail}) + (15+4\text{ compensate}) = \mathbf{60\text{ ms}}$ |
| Step 3 Failure (Payment declined) | Happy path steps (1 & 2) + Payment (declines) + Inventory release (comp) + Order cancellation (comp) | $298 + (15+6\text{ comp2}) + (15+4\text{ comp1}) = \mathbf{338\text{ ms}}$ |
Suppose your checkout service handles a surge of 200 checkouts/second. Your outbox publisher process polls the database every 100ms. What happens if the outbox publisher encounters network lag that limits its message publishing rate to 150 messages/second?
- Compute backlog accumulation rate: $$Backlog_{rate} = 200 \text{ checkouts/s} - 150 \text{ published/s} = 50 \text{ messages/s}$$
- Compute backlog after a 5-minute (300s) network degradation: $$Backlog_{total} = 50 \text{ messages/s} \times 300 \text{ seconds} = 15,000 \text{ events}$$
- Compute catch-up time once network recovery allows publishing at 300 messages/second: Once recovery occurs, the net publishing speed is $300 - 200 \text{ incoming} = 100 \text{ messages/s}$. $$T_{catch\_up} = \frac{15,000 \text{ events}}{100 \text{ events/s}} = 150 \text{ seconds (2.5 minutes)}$$
This illustrates why outbox queues must use high-throughput database partitioning and why you need monitoring alerts for outbox queue lag.
How to debug & inspect it
To inspect and audit distributed Sagas, trace the workflow identifiers across systems and verify that the outbox table events are being drained properly.
Use the guide below to resolve common failures in Saga-based checkout architectures:
| Symptom | Likely Cause | Fix |
|---|---|---|
| Duplicate orders created when clients retry checkout due to network timeouts | Lack of idempotency check at the Orchestrator or Order Service layer | Enforce a unique constraint on idempotency_key in the database; return cached response on match. |
| "Ghost Inventory Lock": items remain locked in the warehouse, but no order was completed | A compensating transaction (release) failed to execute or crashed mid-rollback | Implement dead-letter queues (DLQ) for failed rollbacks; orchestrator must retry compensating steps infinitely. |
| The transactional outbox table grows indefinitely, causing slow DB queries | Drained outbox rows are not being deleted or archived | Configure the outbox publisher to prune successfully published rows immediately, or partition the table. |
🧠 Quick check
1. Why is Two-Phase Commit (2PC) avoided for high-throughput microservice checkouts?
Two-Phase Commit relies on a coordinator holding global locks on database tables. If one service experiences network latency, all other services' resources remain locked, dragging down system availability.
2. What is the defining characteristic of a "compensating transaction" in a Saga?
A compensating transaction cannot simply perform a database "rollback" because the local transaction was already committed. Instead, it runs an explicit business undo operation (e.g. issuing a refund API call).
3. What failure mode does the Transactional Outbox pattern prevent?
By writing both the business record and the outbox event payload inside the same local database transaction, you guarantee that either both are saved or both are rejected, preventing data drift.
4. In Saga Orchestration, where does the execution state and logic live?
Saga Orchestration uses a centralized coordinator service (orchestrator) that acts as a state machine. It is responsible for calling API endpoints, tracking step verdicts, and executing rollbacks on failure.
✍️ Exercise: design the compensation outbox schema
A customer initiates checkout. The orchestrator reserves inventory (Step 1) and charges payment (Step 2). While trying to create the order in the Order Database (Step 3), the Order DB is down. The orchestrator must roll back.
Design the Outbox table schema for the Orchestrator Database itself to ensure that the compensating rollback (Release Inventory) is guaranteed to execute even if the orchestrator server crashes mid-rollback.
Model answer:
To guarantee that compensating transactions are never lost during orchestrator crashes, the orchestrator must persist the saga's state and pending rollback tasks in its local database before attempting any network calls.
Orchestrator Saga Job Table Schema (SQL):
CREATE TABLE saga_executions (
saga_id VARCHAR(64) PRIMARY KEY,
status VARCHAR(20) NOT NULL, -- 'STARTED', 'COMPENSATING', 'FAILED', 'SUCCESS'
current_step VARCHAR(50) NOT NULL,
payload JSONB NOT NULL, -- Holds checkout parameters
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);
CREATE TABLE saga_compensating_tasks (
task_id VARCHAR(64) PRIMARY KEY,
saga_id VARCHAR(64) REFERENCES saga_executions(saga_id),
target_service VARCHAR(50) NOT NULL, -- 'INVENTORY_SERVICE', 'PAYMENT_SERVICE'
payload JSONB NOT NULL, -- Parameters for compensating API
status VARCHAR(20) NOT NULL, -- 'PENDING', 'SENT', 'COMPLETED'
retry_count INT DEFAULT 0,
next_retry_at TIMESTAMP NOT NULL
);
Rollback Execution Logic:
- When Step 3 fails, the orchestrator updates
saga_executions.status = 'COMPENSATING'and inserts a task insaga_compensating_taskswithstatus = 'PENDING'. This happens inside a single SQL transaction on the orchestrator DB. - An asynchronous background worker polls the
saga_compensating_taskstable forPENDINGrows, makes the REST call to Inventory Service/v1/inventory/release, and updates status toCOMPLETEDupon receipt of 200 OK. - If the orchestrator crashes, the next spawned orchestrator node scans the task table, finds the unfinished task, and retries it. This ensures eventual consistency despite server crashes.
Key takeaways
- **Avoid 2PC at scale**. Two-Phase Commit holds resources hostage. Use the Saga pattern for asynchronous eventual consistency across microservice databases.
- **Orchestration vs. Choreography**. Choose Orchestration when workflows have many steps, loops, or complex rollback paths; it centralizes logging and visibility.
- **All steps must be reversible**. Sagas rely on compensating transactions. For every endpoint that mutates state, design a corresponding endpoint to reverse it.
- **Outbox prevents drift**. Use the Transactional Outbox pattern to write business records and outgoing messages atomically to the database, ensuring at-least-once delivery.
- **Retries must be infinite for rollbacks**. A failed normal step triggers a rollback. Rollback actions must be retried until they succeed; otherwise, data drifts permanently.
Sources & further reading
- Chris Richardson — Microservices Patterns: The Saga Pattern — the core reference design for orchestration and choreography sagas
- Chris Richardson — Transactional Outbox Pattern — details on avoiding distributed state drift
- Debezium CDC Architecture — how change data capture tailing DB write-ahead logs handles outbox publishing