Reliability & Scale · Lesson 18

Resilience & disaster recovery

A system that either works perfectly or falls over completely is not resilient — it is brittle. The goal is never zero failures; it is ensuring that when a dependency breaks, the rest of the system keeps moving at reduced capacity rather than stopping entirely.

The difference between availability and resilience

High availability is about surviving the failure of an infrastructure component — a crashed server, a flipped AZ, a misconfigured load balancer. Resilience is a different and complementary goal: surviving the misbehaviour of a dependency that is technically still alive. A downstream service that responds in 8 seconds instead of 80 milliseconds is not down — but a caller that holds a thread open waiting for it is being quietly strangled. Resilience patterns are the bulkheads, pressure relief valves, and blast walls that prevent one struggling component from sinking everything around it.

The underlying principle is graceful degradation: when a dependency is unavailable or overloaded, return a degraded but useful response rather than an error. Serve a cached product list when the inventory service is slow. Show a notification count of "99+" when the count service is down rather than failing the page load. Omit the recommendations widget when the ML service times out. Each of these is worse than the full experience — and each is vastly better than a blank error screen.

Bulkheads: containing blast radius

The name comes from ship design. A bulkhead is a watertight wall between compartments of a hull. When one compartment floods, the walls contain the water. The ship loses some buoyancy but does not sink. In software, a bulkhead is a resource isolation boundary that prevents one slow or failing dependency from consuming every connection, thread, or memory allocation in the process.

The most concrete implementation is separate connection pools per downstream dependency. Suppose a service has three dependencies: a primary database, a recommendations service, and a notification service. Without bulkheads, all outbound I/O shares one system-level connection pool or thread pool. If the recommendations service starts responding in 10 seconds, threads pile up waiting for it. Soon the pool is exhausted — every request, including those that never touch recommendations, queues behind waiting threads. The primary database calls, which were fast, can no longer get a thread. The entire service becomes unresponsive because of a component it relies on for one optional feature.

With bulkheads: a pool of 20 threads handles recommendations. A separate pool of 50 handles primary database calls. The notification pool has 10. When recommendations stalls, it fills its own 20-thread pool and begins rejecting new requests to that dependency. The primary database pool is untouched. The service degrades (no recommendations) but continues serving its core function.

Load shedding: choosing which work to drop

When a service is overloaded it has three options: accept all incoming requests and serve them slowly (adding to the queuing problem), crash, or deliberately reject a fraction of incoming work to protect its ability to serve the rest. Only the third is resilient. Load shedding is the deliberate act of returning a fast error — typically HTTP 503 or 429 — to some requests so that others complete successfully.

Stripe's engineering team has written publicly about their load shedding approach (see sources). The key insight is that not all requests have equal priority. A request to process a payment must be attempted even under extreme load. A request to fetch the account dashboard can be dropped — the user gets a "try again" message, which is frustrating but not catastrophic. Load shedding works by classifying requests by priority at the edge and returning 503 to lower-priority traffic once a configurable concurrency threshold is exceeded. See Rate Limiting Algorithms (Lesson 03) for the algorithmic primitives (token bucket, concurrency limiters) used to implement the concurrency threshold.

Backpressure: pushing the signal upstream

Load shedding says "I am too busy; go away." Backpressure is a related idea applied to internal queues and streaming systems: rather than accepting work into an unbounded queue and hiding the overload, the system signals congestion to the sender so the sender can slow down, instead of the queue growing without bound.

In a producer-consumer pipeline, the concrete mechanism is a bounded queue with a finite capacity. When the queue is full, the producer's call to enqueue() blocks or returns an error immediately, instead of succeeding. The producer receives immediate signal that the consumer is not keeping up. It can either shed that unit of work (drop the message, return an error to its caller) or apply backpressure upstream — block its own caller until the queue drains. This is how TCP flow control works: when the receive buffer fills, the TCP window shrinks to zero, stopping the sender at the transport layer. Designing application-level queues the same way prevents the classic failure mode where a slow consumer causes unbounded memory growth until the process OOMs and crashes.

Timeouts everywhere

Every outbound network call must have a timeout. A missing timeout converts a slow dependency into a thread that waits indefinitely, which is the entry point to the bulkhead failure described above. "Everywhere" is not rhetorical: the HTTP client to the upstream service, the database connection, the cache client, the DNS lookup inside the HTTP client, and the TCP connection establishment all need individual timeouts. Default settings for many HTTP client libraries are either infinite (no timeout at all) or dangerously long (60+ seconds). Set them explicitly at a level consistent with your SLO.

Timeouts should be paired with retries and circuit breakers. A single timeout → immediate retry is often correct for transient network glitches, but must be bounded: see Retries & Exponential Backoff (Lesson 05) for why unbounded retries amplify load on a struggling service. When a dependency is persistently slow or failing, the circuit breaker pattern — see Circuit Breaker Pattern (Lesson 06) — stops calling it altogether for a recovery window, which both protects the dependency and frees your threads immediately.

Disaster recovery: RTO and RPO

Resilience patterns protect against transient and partial failures. Disaster recovery (DR) addresses a different scenario: the data is gone, or the entire system is destroyed, and you must rebuild. Two numbers define the envelope of that rebuild:

Recovery Time Objective (RTO) is the maximum elapsed time from the moment of a disaster until the service is restored to full operation. RTO is a contractual commitment about speed of recovery. An RTO of 4 hours means users can expect service to resume within 4 hours of a declared disaster — and the company has committed to investing whatever infrastructure is necessary to meet that. An RTO of 15 minutes implies active hot-standby infrastructure; an RTO of 24 hours may allow cold restore from nightly backups.

Recovery Point Objective (RPO) is the maximum amount of data loss the business can tolerate, measured in time. An RPO of 1 hour means you are prepared to lose up to one hour of transactions. If a disaster occurs at 14:37 and your last good backup is from 13:45, you have lost 52 minutes of data — inside your RPO. An RPO of zero means no data loss is acceptable, which requires synchronous replication (writing is not acknowledged until it is confirmed on the replica). An RPO of 24 hours allows nightly backups with no real-time replication.

The lower the RTO and RPO, the higher the infrastructure cost. A zero-RPO system must acknowledge every write only after it has been confirmed on a synchronous replica — which adds the network round-trip to the replica to every write latency. This is the same consistency–availability tension from Consistency & the CAP theorem (Lesson 16), expressed in DR terms.

Backups and restore drills

A backup that has never been restored is a hypothesis, not a guarantee. The most common disaster-recovery failure mode is not the absence of backups; it is discovering during an actual incident that the restore procedure is broken, the backup files are corrupt, or the process takes twice as long as the RTO allows. The only fix is to run restore drills on a schedule — quarterly at minimum — and to treat a successful restore as a measurable metric, not a one-time event.

Backup strategy must match the RPO. Point-in-time recovery (PITR) — continuous archival of write-ahead log segments, as supported by PostgreSQL and Amazon RDS — allows restoring to any second within the retention window, not just the last snapshot. For an RPO of 5 minutes, PITR is the right tool; an hourly snapshot leaves a 55-minute window of potential loss.

Runbooks are as important as the infrastructure. A runbook is a step-by-step operating procedure for a specific failure scenario: what to verify, what commands to run, in what order, and how to confirm the restore succeeded. The runbook exists so that an on-call engineer who has never personally performed a full restore can do so at 3 AM without improvising. Keep runbooks version-controlled, link them to the relevant monitoring alerts, and review them after every drill that surfaces a gap.

Chaos engineering: proving resilience by breaking things

Code review and load testing check that a system behaves correctly under expected conditions. Chaos engineering — deliberately injecting failures into a running system — checks that the resilience patterns actually fire as designed. Netflix's Chaos Monkey is the most famous example: it terminates random production instances to verify that the rest of the fleet handles the loss gracefully, and to surface any implicit assumptions about specific instances always being available (see sources).

Chaos experiments are not random destruction. A well-designed experiment has a hypothesis ("if the recommendations service becomes unavailable, the homepage still loads and shows a static fallback"), a defined blast radius (only one of five recommendation servers, or only in a canary deployment group), and measurable criteria for success. The goal is to find failures in a controlled way before a real incident finds them in an uncontrolled way. Common experiments: terminate random instances, inject network latency (e.g., 300 ms added to all DB calls), exhaust a specific connection pool, simulate a slow DNS response, or drop packets between two services.

Start with a hypothesis, run in non-production first, and verify the hypothesis is confirmed before running in production at limited blast radius. Never run chaos experiments during peak traffic hours — the goal is discovery, not self-inflicted incidents.

Graceful degradation decision flow

Under the hood: bulkhead execution — a traced example

Here is exactly what happens inside a Java or Go service that uses separate semaphore-based connection pools as bulkheads. The mechanism is a counting semaphore: each pool has a max-concurrency value. Each outbound call must acquire a permit before sending the request. When all permits are held by in-flight calls, new callers either block for a maxWait duration or receive an immediate rejection. This is the moment that contains the blast radius — the caller can immediately return the fallback instead of hanging.

// Conceptual bulkhead using a semaphore (Go pseudocode)
type Bulkhead struct {
    sem    chan struct{}  // buffered channel as a counting semaphore
    timeout time.Duration
}

func NewBulkhead(maxConcurrency int, timeout time.Duration) *Bulkhead {
    return &Bulkhead{
        sem:     make(chan struct{}, maxConcurrency), // capacity = max permits
        timeout: timeout,
    }
}

func (b *Bulkhead) Execute(ctx context.Context, fn func() (interface{}, error)) (interface{}, error) {
    // Attempt to acquire a permit within the timeout
    select {
    case b.sem <- struct{}{}:  // slot acquired
        defer func() { <-b.sem }()  // release on return
        return fn()
    case <-time.After(b.timeout):
        // Bulkhead saturated — return immediately, never hang
        return nil, ErrBulkheadFull  // caller routes to fallback
    case <-ctx.Done():
        return nil, ctx.Err()
    }
}

// Usage: separate bulkheads per dependency
var (
    dbBulkhead  = NewBulkhead(50, 100*time.Millisecond)   // primary DB
    recBulkhead = NewBulkhead(20,  50*time.Millisecond)   // recommendations
    ntfBulkhead = NewBulkhead(10,  50*time.Millisecond)   // notifications
)

// In the request handler:
result, err := recBulkhead.Execute(ctx, func() (interface{}, error) {
    return fetchRecommendations(userID)
})
if errors.Is(err, ErrBulkheadFull) {
    recommendations = defaultRecommendations()  // fallback — never blocks
}

The trace of the failure scenario from the diagram above:

The key observation: the database pool's in_flight counter stays well below its capacity throughout. The degradation is contained entirely within the recommendations pool. Response times for pages that never touch recommendations are unaffected.

How to operate it: cascading-failure triage and RTO/RPO worked example

Cascading-failure triage table

RTO / RPO worked example

RPO = 2 hours. Nightly snapshots (24-hour RPO) do not meet this. The right tool is continuous WAL archival to S3 (PITR on Amazon RDS), which allows restore to any point within the retention window. With PITR, worst-case data loss is the time since the last archived WAL segment, typically under 5 minutes — well inside the 2-hour RPO.

RTO = 30 minutes. A cold restore from an S3 backup to a new RDS instance typically takes 15–45 minutes depending on database size. For a 30-minute RTO, the warm-standby approach is safer: keep a pre-provisioned standby instance that continuously receives replicated data and can be promoted in under 2 minutes (Amazon RDS Multi-AZ achieves this). The standby instance runs in a different AZ and is available for promotion without requiring a full restore from backup.

Architecture decision. Enable RDS Multi-AZ (provides RTO of ~2 minutes, RPO of ~0 seconds for in-region failures) and continuous PITR with a 7-day retention (protects against accidental data deletion, logical corruption, and events that damage both AZ copies). Run a restore drill quarterly: restore from PITR to a new RDS instance, verify row counts and referential integrity, measure the elapsed time, and update the runbook.

By the numbers

Make it concrete. The service is an e-commerce checkout API. It calls a Recommendations dependency at 500 QPS; each call takes on average 200 ms. The checkout database holds 500 GB and is backed up to S3. Workers are CPU-bound and run at roughly 70% utilisation during normal traffic.

Bulkhead pool sizing: Little's Law

The right pool size for a dependency equals the peak number of concurrent in-flight calls — anything larger wastes memory; anything smaller starves throughput. From Little's Law (L = λ × W):

pool_size = QPS_dep × latency_dep (in seconds)
          = 500 req/s × 0.200 s
          = 100 concurrent connections

Cap the Recommendations pool at 100. When the dependency slows from 200 ms to 2,000 ms under load, the same formula tells you the pool would need 500 × 2.0 = 1,000 slots — but by capping at 100, new calls fail fast after maxWait = 50 ms and return the fallback. The primary DB pool (capped separately at 50) is never touched. Stripe's public architecture (Stripe Engineering — Scaling your API with rate limiters) uses exactly this pattern: separate concurrency limiters per dependency type.

RPO: backup interval determines maximum data loss

RPO = backup interval for snapshot-based backups. With hourly snapshots, a disaster at 14:58 hits against the 14:00 snapshot — up to 58 minutes of data loss. To meet an RPO of 5 minutes, you need continuous WAL archival (PITR), where each WAL segment is shipped to S3 every 60–300 seconds, limiting loss to that shipping interval (PostgreSQL PITR documentation).

RTO: how long does a restore actually take?

The checkout database is 500 GB. A cold restore from S3 to a new RDS instance at AWS S3-to-RDS throughput of roughly 100 MB/s takes:

restore_time = database_size ÷ restore_throughput
             = 500 GB × 1,024 MB/GB ÷ 100 MB/s
             = 512,000 MB ÷ 100 MB/s
             ≈ 5,120 s  ≈  85 minutes

An 85-minute cold restore cannot meet a 30-minute RTO. The fix is a warm standby (RDS Multi-AZ), where promotion — not full restore — takes 28–60 seconds. Use PITR as the safety net for logical corruption; use the warm standby for RTO. Both run together (AWS RDS PITR docs).

Load-shedding threshold: when to start dropping work

Workers run at 70% CPU utilisation at normal load. The load-shedding threshold is set at 85% worker utilisation. Below 85%: all requests proceed. At or above 85%: low-priority requests (recommendations, dashboard analytics) receive HTTP 503 immediately; high-priority requests (checkout, payment) continue. This protects the 15% headroom needed for retries and GC pauses (AWS Builders' Library — load shedding).

headroom = 1 − shed_threshold = 1 − 0.85 = 15%
normal_load = 70% utilisation → 15% gap before shedding triggers
at 85% shed begins → high-priority work still completes → system stays stable

Decision math: choosing pool size, backup interval, and RTO feasibility

Given dependency QPS and latency, the formula is: pool_size = QPS × latency_s. Round up to the nearest 10 and add 10% headroom for burst. For RPO: pick the backup method whose interval fits inside the RPO. For RTO: compare database_size ÷ restore_rate against the RTO target — if it exceeds it, you need a warm standby, not cold restore. For load shedding: set the threshold at 1 − GC_headroom − retry_headroom, typically 80–85%.

Sources & further reading

• AWS Well-Architected Framework – Reliability Pillar
• Google SRE Book – Chapter 22: Addressing Cascading Failures
• Stripe Engineering – Scaling your API with rate limiters (load shedding section)
• Netflix Tech Blog – The Netflix Simian Army (Chaos Monkey)
• Principles of Chaos Engineering
• AWS Docs – Restoring a DB instance to a specified time (PITR)