API Reliability · Lesson 19

Reliability: Chaos & Progressive Delivery

Writing code is only half the battle; deploying it safely under heavy load without degrading user experience is what defines a resilient production API. We will deep-dive into progressive rollouts (canaries and blue-green deploys), connection draining strategies during rolling updates, and using chaos engineering fault injection to proactively test resilience limits before outages occur.

1. Progressive Delivery: Blue-Green vs. Canaries

Exposing a brand-new API version to 100% of production traffic at once is high risk. **Progressive Delivery** mitigates this by shifting traffic incrementally, allowing operations teams to observe performance signals (latencies, error rates) before committing to a full release.

Canary Shifting Keys

• Random Percentage: Shifting exactly 5% of all connections. (Good for general load testing).
• User/Account Ring: Directing traffic based on headers (e.g. routing internal employees first, then beta users, then general public).
• Geo-routing: Deploying the new version to a single small geographic region first (e.g., EU-West-1) before rolling it out globally.

2. Graceful Shutdown & Connection Draining

During a rolling deployment, old containers/servers must be destroyed and replaced by new ones. If you kill the old process immediately, active HTTP requests will terminate mid-flight (returning `502 Bad Gateway` or connection resets to users), and WebSocket connections will disconnect abruptly.

**Graceful shutdown** forces the application server to handle terminating signals cleanly: stop accepting *new* connections, finish processing *active* in-flight requests, and only then terminate the process.

Under the hood: Canary routing & connection draining

This diagram traces how a load balancer routes traffic progressively, shifting 5% to the canary nodes, and how old nodes drain connections gracefully upon receiving a SIGTERM signal.

3. Chaos Engineering: Fault Injection

Resilience patterns like circuit breakers, retries, and rate limiters look excellent in architecture diagrams, but they often fail silently in production because they are rarely exercised. **Chaos Engineering** (popularized by Netflix's Chaos Monkey) proactively injects real failures into staging or production systems to confirm that client boundaries degrade gracefully.

Common Chaos Injection Vectors

• Latency Injection: Artificially delaying requests by a random interval (e.g. $+3000\text{ ms}$) to verify that client timeouts and circuit breaker open states trigger correctly.
• Error Injection: Forcing a specified percentage (e.g., 5%) of database queries or API routes to return `503 Service Unavailable`.
• Resource Exhaustion: Artificially eating up CPU cores or filling up network cards to test autoscale trigger parameters.

By the numbers: canary statistical significance math

When running a canary deployment, how long must you wait to be statistically confident that the new version is not introducing more errors than the old version?

Governing Equations

• Required Sample Size (A/B Test / Z-Test for Proportions): To detect a difference in error rate between control version $p_1$ and canary version $p_2$ with statistical confidence, the minimum number of requests $N$ required per group is: $$N = \frac{(Z_{\alpha/2} + Z_\beta)^2 \cdot [p_1(1-p_1) + p_2(1-p_2)]}{(p_1 - p_2)^2}$$ Where:
  • $Z_{\alpha/2}$ is the significance threshold (for $95\%$ confidence / type-I error rate $\alpha=0.05$, $Z_{\alpha/2} = 1.96$).
  • $Z_\beta$ is the statistical power threshold (for $80\%$ power / type-II error rate $\beta=0.20$, $Z_\beta = 0.84$).
• Required Canary Duration: $$Duration_{minutes} = \frac{N}{QPS_{canary} \times 60}$$

Scenario Parameters

• Baseline Production Error Rate ($p_1$): 0.1% (0.001)
• Minimum Detectable Error Rate Increase ($p_2$): 0.3% (0.003) (we want to trigger rollback if errors triple)
• Significance ($\alpha$): 0.05 (95% confidence)
• Power ($1-\beta$): 0.80 (80% power)
• Total Production Traffic: 10,000 QPS
• Canary Allocation: 2% (200 QPS)

Worked Calculations: Canary Evaluation Window Sizing

1. Compute statistical constants: $$(Z_{\alpha/2} + Z_\beta)^2 = (1.96 + 0.84)^2 = 7.84$$
2. Compute numerator variance: $$Variance = 0.001(0.999) + 0.003(0.997) \approx 0.001 + 0.00299 = 0.00399$$
3. Compute the denominator effect size: $$(p_1 - p_2)^2 = (0.001 - 0.003)^2 = (-0.002)^2 = 0.000004$$
4. Calculate required requests $N$: $$N = \frac{7.84 \cdot 0.00399}{0.000004} = \frac{0.03128}{0.000004} = \mathbf{7,820 \text{ requests}}$$ We need at least 7,820 requests to hit the canary version to prove the error rate delta is real rather than random noise.
5. Compute required runtime duration: With a 2% canary allocation receiving 200 QPS: $$Duration_{seconds} = \frac{7,820}{200\text{ QPS}} = 39.1\text{ seconds}$$ This is an excellent result! At 10,000 production QPS, we can safely validate the canary health in **under 1 minute** of observation.

How to debug & inspect it

Observe graceful shutdown processes and test failure routes using shell output and header parameters.

Review standard configurations for progressive delivery controls below:

class GracefulServer {
    active_requests_count = 0
    is_shutting_down = false
    
    function start() {
        register_signal_handler("SIGTERM", handle_sigterm)
    }
    
    function handle_request(req, res) {
        if is_shutting_down:
            res.set_status(503) // Tell LB/Client to try another server
            res.write("Server is shutting down")
            return
            
        active_requests_count += 1
        try {
            process_request(req, res)
        } finally {
            active_requests_count -= 1
        }
    }
    
    function handle_sigterm() {
        is_shutting_down = true
        logger.info("SIGTERM received. Starting connection draining.")
        
        // Let the load balancer detect 503s or health-check failure to detach us
        sleep(5000) 
        
        drain_timeout = 30 // seconds max to wait
        start_time = current_time()
        
        while active_requests_count > 0:
            if current_time() - start_time > drain_timeout:
                logger.warn("Force shutdown: drain timeout reached with active requests remaining")
                break
            sleep(0.5) // Wait for requests to finish
            
        logger.info("All requests drained. Exiting.")
        system_exit(0)
    }
}

Sources & further reading

• Principles of Chaos Engineering — the foundation documentation for chaotic testing
• Kubernetes Pod Termination Lifecycle — how container platforms handle SIGTERM and connection draining
• Netflix Technology Blog — Resiliency & Chaos at Scale