Reliability & Scale · Lesson 17

High availability & redundancy

Availability is not something you bolt on after a system works — it is a structural property determined before you write a line of code. Every tier that has exactly one instance is a ticking clock.

Why one of anything is never enough

A restaurant that employs a single chef can produce excellent food — right up until the moment that chef calls in sick. On an ordinary Tuesday that risk feels theoretical. On a Friday night with a full reservation book it is a catastrophe. The restaurant's entire revenue-generating capability depends on one person who can fail.

The same logic applies to every component in a system. A single application server, a single database instance, a single load balancer — each one is a single point of failure (SPOF): a component whose failure causes the entire system to stop serving users. Redundancy is the practice of running multiple independent copies of each component so that one failing does not stop the others.

The critical insight is that redundancy must be applied at every tier independently. Deploying two application servers behind one load balancer does not make you highly available if the load balancer itself has no redundant peer. Eliminating SPOFs requires a systematic audit: compute, database, cache, load balancer, DNS, and the network paths between them.

Availability Zones vs Regions

Cloud providers organise their infrastructure into two nested scopes that matter for redundancy design.

An Availability Zone (AZ) is a physically separate datacenter — its own power grid, cooling plant, and network gear — located within a metropolitan area. AZs within the same region are connected to each other by low-latency private fiber (typically sub-2ms round-trip). The key word is physically separate: a power outage, hardware fire, or network failure in one AZ does not propagate to its siblings, because they share no physical infrastructure.

A Region is a geographically distinct cluster of AZs — for example, AWS us-east-1 contains six AZs spread across northern Virginia. Regions are separated by hundreds or thousands of kilometres. A single Region failing is rare and requires a large-scale event: a natural disaster, a major internet exchange outage, or a catastrophic DNS misconfiguration. When a Region goes down, recovery requires traffic to be routed to an entirely different Region.

For most services, multi-AZ is the right baseline. Multi-region is reserved for services whose SLO cannot be met by multi-AZ alone, or where latency to global users is a primary concern.

Active-active vs active-passive

Once you have redundant nodes, you need a strategy for how traffic is distributed across them — and what happens when one fails.

Active-active means all nodes serve live traffic simultaneously. The load balancer routes requests across every healthy instance. When one node fails, the load balancer's health check detects the failure and stops sending it new connections. No promotion step is needed — the other nodes simply absorb the removed capacity. The Recovery Time Objective (RTO) equals the health check detection window (typically 10–30 seconds). The trade-off: all active nodes must be capable of handling writes, which requires careful consistency management.

Active-passive means one node (the primary) handles all traffic while a standby node waits idle, continuously receiving replicated data but not serving requests. When the primary fails, a failover controller promotes the standby to primary and redirects traffic to it. This promotion step adds 20–60 seconds to recovery for typical managed databases (e.g., Amazon RDS Multi-AZ). The advantage: only one node ever accepts writes, which makes consistency far simpler to reason about.

Automatic failover driven by health checks

Redundancy is inert without a mechanism to detect failure and reroute traffic. That mechanism is the health check. See Load Balancing (Lesson 08) for how load balancers use health checks to remove unhealthy nodes from rotation; this section focuses on what a health check should test.

Kubernetes exposes two probes with distinct jobs:

• Readiness probe — answers the question: is this container ready to accept traffic? A failing readiness probe removes the pod from the Service's endpoint list (takes it out of LB rotation) without restarting the container. Use this during warm-up, database migrations, or any state where the process is running but not yet safe to receive requests.
• Liveness probe — answers the question: is this container fundamentally alive, or should it be restarted? A failing liveness probe causes kubelet to kill and restart the container. Use this to recover from deadlocks or states the process cannot self-heal from.

Getting these backwards is a common mistake. Configuring a failing readiness condition as a liveness probe causes cascading restarts instead of graceful traffic shedding — every pod that is momentarily overloaded gets killed, making the overload worse.

# Kubernetes pod spec — both probes on a typical API server
livenessProbe:
  httpGet:
    path: /health/live    # returns 200 if process is alive; 503 to trigger restart
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3     # 3 consecutive failures → restart

readinessProbe:
  httpGet:
    path: /health/ready   # checks DB conn, cache, downstream deps
    port: 8080
  periodSeconds: 5
  failureThreshold: 2     # 2 failures → remove from LB rotation
  successThreshold: 1     # 1 success → return to rotation

The health check's failure threshold and period together determine your detection window: with periodSeconds: 5 and failureThreshold: 3, the load balancer waits up to 15 seconds before removing a node. Tune these against your SLO's error budget — the detection window is unavoidable downtime for users routed to the failing instance.

Multi-region

Multi-AZ protects against a single datacenter failing. For events that affect an entire metropolitan area — a regional internet exchange failure, a natural disaster, a large-scale cloud-provider incident — you need traffic to route to an entirely different region.

Multi-region deployments provide two distinct benefits:

• Lower latency: Serve users from the nearest region. A user in Singapore routed to a US-East region experiences 200–250 ms of base latency before a byte arrives. Routing them to a Singapore region drops that to single-digit milliseconds.
• Disaster survival: If one region becomes unavailable, traffic fails over to another. Users see elevated latency but the service remains reachable.

Both benefits come with costs. Data written in one region must be replicated to other regions, introducing replication lag. Depending on consistency requirements, reads from a secondary region may see stale data — the same tension explored in Consistency & the CAP theorem (Lesson 16). Multi-region active-active is the hardest configuration to operate: writes must be globally replicated, conflicts must be detected and resolved, and the system must handle network partitions between regions gracefully.

Mapping the nines to architecture

An availability target of "99.9%" is not a single architectural decision — each additional nine requires a qualitatively different architecture, not just more instances. The nines encode structural requirements:

These budgets assume failures are independent. If your database and your application server share the same AZ and the AZ fails, you've consumed both components' downtime simultaneously. This is why multi-AZ must be applied to every tier — a multi-AZ application server cluster backed by a single-AZ database still has a single-AZ SPOF.

Tie availability targets to your error budget framework — see SLIs, SLOs & SLAs (Lesson 10). The error budget is the arithmetic expression of the nines, and it is what your team burns through during an incident.

Multi-AZ and multi-region topology

Under the hood: a failover traced second by second

Abstract descriptions of failover hide the details that matter operationally. Walk through exactly what happens when AZ-1's application server stops responding in a multi-AZ active-passive setup:

1. t = 0 s — Instance stops responding. The application server in AZ-1 crashes (kernel panic, OOM kill, power failure). Existing in-flight HTTP connections hang. New connection attempts time out.
2. t = 0–15 s — Health probes time out. The load balancer sends HTTP GET /health/ready probes at 5-second intervals. All three return no response (connection refused or timeout). Each failure increments the LB's unhealthy threshold counter.
3. t = 15 s — LB removes the instance from rotation. After three consecutive failures (threshold reached), the LB marks the instance unhealthy and stops routing new connections to it. Existing connections that were mid-flight get a TCP RST from the LB or time out on the client side.
4. t = 15 s+ — AZ-2 absorbs all traffic. Every new request goes to healthy AZ-2 app servers. Users experience elevated latency (AZ-2 may be slightly more loaded) but requests succeed.
5. t = 16 s — DB failover initiates. If active-passive: the DB failover controller (or AWS RDS Multi-AZ) detects the primary in AZ-1 is unreachable. It begins promoting the AZ-2 standby to primary. This promotion takes 20–60 seconds for RDS.
6. t = 45 s — DNS TTL expires; new connections resolve to the promoted standby. RDS updates the CNAME record for the DB endpoint to point to the new primary. Clients whose DNS TTL has expired re-resolve and connect to AZ-2's new primary. Clients that held connections to the old primary receive TCP RST and must reconnect. Connection pool libraries typically handle this transparently with retry logic.

Total RTO in this scenario: roughly 45–60 seconds. The error budget consumed depends on how many requests were in-flight during the 15-second detection window and whether connection pools retried transparently.

Failover timeline

How to operate it: symptom to fix

By the numbers

Make it concrete. The service is a B2B SaaS API targeting 99.99% annual availability (four nines). Every architectural decision below is evaluated against that budget.

Series vs. redundancy: the availability math

When components are in series (all must be up for the service to work), their availabilities multiply — and the product is always worse than the weakest component:

A_series = a₁ × a₂ × … × aₙ

3 components each at 99.9%:
A_series = 0.999 × 0.999 × 0.999 = 0.997 = 99.7%   ← WORSE than any individual component

The lesson: a three-tier stack (load balancer + app server + database) that runs only one instance of each tier is limited to 99.7% even if every component is individually 99.9%. Redundancy is the fix.

When components are redundant (any one of k instances can fail and the others take over), the combined availability is:

A_redundant = 1 − (1 − a)^k      (assumes independent failures)

Two 99% instances (k = 2):
A_redundant = 1 − (1 − 0.99)² = 1 − 0.0001 = 0.9999 = 99.99%

Two mediocre 99% instances in redundancy deliver better availability than one excellent 99.9% instance alone. This is why redundancy is the primary lever, not individual component reliability.

Nines → downtime/year table

Memory aid: each additional nine cuts the annual downtime budget by 10×. Going from three nines to four nines shrinks your budget from 8.76 hours to 52.6 minutes — a single careless deployment or a 45-minute database failover can consume the entire year's allowance in one incident.

Failover budget: does it fit the SLO?

For a four-nines target (52.6 min/year), trace the exact failover timing for an active-passive multi-AZ setup with periodSeconds: 5 and failureThreshold: 3:

Does it fit? The annual budget for 99.99% is 52.6 minutes = 3,156 seconds. One incident burns 55 s, so you can sustain at most ⌊3,156 ÷ 55⌋ = 57 incidents per year (about one per week) before breaching the SLO. That seems comfortable — but any incident with manual steps, a slow DNS TTL, or a multi-step runbook that takes 5 minutes instead of 55 seconds would consume 5.5 incidents' worth of budget in a single event.

Decision math: how many redundant instances to hit a target

You need each tier to contribute negligible failure probability. Solving A_redundant ≥ A_target for k:

k ≥ log(1 − A_target) ÷ log(1 − a)

Target A = 99.99%, component a = 99.9%:
k ≥ log(1 − 0.9999) ÷ log(1 − 0.999)
  = log(0.0001) ÷ log(0.001)
  = (−4) ÷ (−3)
  ≈ 1.33  →  round up to k = 2 instances per tier

Two 99.9% instances per tier (compute + database + load balancer) combine for: A = 1 − (0.001)² = 99.9999% per tier, well above the 99.99% target. The series product of three tiers is then 0.999999³ ≈ 99.9997% — still above four nines. This is why multi-AZ active-passive with two nodes per tier is the standard four-nines architecture.

Sources

• AWS Well-Architected Framework – Reliability Pillar
• Google SRE Book – Chapter 20: Load Balancing at the Frontend
• Google SRE Book – Chapter 26: Data Integrity
• AWS Docs – RDS Multi-AZ
• Kubernetes – Configure Liveness, Readiness and Startup Probes