Reliability & Scale · Lesson 06

Circuit Breaker Pattern

When a downstream service is drowning, the last thing it needs is your service pouring more requests on top. The circuit breaker is an automatic trip-switch: it detects the failure, cuts the flow, and only restores it once the downstream has had a chance to breathe.

The problem: hammering a drowning neighbor

Picture a neighborhood sharing a single water main. One house springs a burst pipe; the water pressure in the whole street drops. Every other house starts running all their taps harder, trying to compensate — which drops the pressure even more. Now the problem has spread from one burst pipe to the entire street.

Microservices fail the same way. Service A calls Service B. B is overloaded and starts responding slowly. A's request threads pile up waiting for B to answer. A's own response times balloon. Services C and D that call A now start timing out. One slow dependency has metastasized into a full cascading failure — not because B was catastrophically broken, but because everyone kept hammering it instead of backing off.

This is the problem the circuit breaker was designed to solve. The name is borrowed directly from electrical engineering: a physical circuit breaker trips when current exceeds a safe threshold, protecting the wiring. In software, a circuit breaker trips when a downstream's error rate exceeds a threshold, protecting threads, connections, and users.

The three states

A circuit breaker wraps every call to an external dependency and maintains a small state machine with three states:

Closed

The normal operating state. All calls pass through to the dependency. The breaker counts successes and failures inside a rolling time window (e.g., the last 60 seconds). As long as the failure rate stays below a configured threshold (e.g., 50%), nothing changes.

Open

The breaker has tripped. Calls are rejected immediately — the dependency is not contacted at all. This "fail fast" behavior is crucial: instead of tying up threads waiting for a service that can't respond, the caller gets an instant error and can apply a fallback. The dependency simultaneously gets zero additional load, giving it room to recover.

Half-Open

After a configured cool-down period (e.g., 30 seconds), the breaker enters a cautious probe state. It allows exactly one (or a small fixed number of) request(s) through to the real dependency. If the probe succeeds, the circuit closes and normal traffic resumes. If the probe fails, the circuit trips back to OPEN and the cool-down restarts. This gradual re-entry prevents immediately flooding a fragile but recovering service.

Configurable thresholds

A production circuit breaker needs at least four tuneable parameters:

The minimum request count matters: if you've only had 2 requests and both failed, that's 100% — but it's not statistically meaningful. Without a floor count, a breaker trips the moment a service boots up and the first two warmup calls are slow.

Fallbacks and graceful degradation

A circuit breaker alone just replaces a slow failure with a fast one. The real value is pairing it with a fallback: an alternative behavior when the dependency is unavailable. Fallback options, roughly from best to worst user experience:

1. Cached data — return the last known-good response if it's still within an acceptable staleness window.
2. Default / empty state — return an empty list, a zero count, or a neutral default rather than an error.
3. Partial response — omit the unavailable section and flag it: { "recommendations": null, "recommendations_available": false }.
4. Queue for later — accept the write and process it once the dependency recovers (for async operations).
5. Clear error message — if no fallback is viable, return a user-friendly message rather than a 500.

The goal is graceful degradation: a service that behaves "less well" during a dependency outage is far preferable to one that becomes completely unavailable.

Pairing with timeouts and bulkheads

A circuit breaker without a timeout is toothless. If calls to the dependency have no deadline, a thread can wait indefinitely — the circuit never accumulates enough failures to trip because most requests are "pending" rather than "failed". Set an explicit connection timeout and read timeout on every outbound call. Typically these are separate: connection timeout 500 ms, read timeout 5 s.

A bulkhead complements this: it limits the number of concurrent calls to a single dependency using a dedicated thread pool or semaphore. If the pool is exhausted, new requests fail fast immediately rather than queuing — containing the blast radius to calls destined for that one dependency, not the entire caller service.

Worked example: minimal circuit breaker

// Pseudo-code: SimpleCircuitBreaker
// Wraps any callable that can fail (e.g., an HTTP client call).

class CircuitBreaker {
  constructor(options = {}) {
    this.failureThreshold  = options.failureThreshold  ?? 0.5;  // 50%
    this.minRequests       = options.minRequests       ?? 10;
    this.windowMs          = options.windowMs          ?? 60_000;
    this.cooldownMs        = options.cooldownMs        ?? 30_000;

    this.state         = "CLOSED";
    this.openedAt      = null;
    this.calls         = [];  // ring-buffer of { ts, success }
  }

  // Call this instead of calling the dependency directly.
  async call(fn, fallback) {
    if (this.state === "OPEN") {
      const elapsed = Date.now() - this.openedAt;
      if (elapsed < this.cooldownMs) {
        // Fail fast — don't touch the dependency
        return fallback ? fallback() : throw new Error("Circuit OPEN");
      }
      // Cool-down expired → probe
      this.state = "HALF_OPEN";
    }

    try {
      const result = await fn();
      this._record(true);
      if (this.state === "HALF_OPEN") {
        this.state = "CLOSED";  // probe passed — reopen traffic
        this.calls = [];
      }
      return result;
    } catch (err) {
      this._record(false);
      if (this.state === "HALF_OPEN") {
        this._trip();  // probe failed — stay OPEN
      }
      throw err;
    }
  }

  _record(success) {
    const now = Date.now();
    // Evict entries outside the rolling window
    this.calls = this.calls.filter(c => now - c.ts < this.windowMs);
    this.calls.push({ ts: now, success });

    if (this.calls.length >= this.minRequests) {
      const failures = this.calls.filter(c => !c.success).length;
      if (failures / this.calls.length > this.failureThreshold) {
        this._trip();
      }
    }
  }

  _trip() {
    this.state    = "OPEN";
    this.openedAt = Date.now();
  }
}

// Usage:
const breaker = new CircuitBreaker({ failureThreshold: 0.5, cooldownMs: 30_000 });

async function getUserProfile(userId) {
  return breaker.call(
    () => profileService.get(userId),          // real call
    () => ({ name: "Unknown", avatar: null })  // fallback
  );
}

Under the hood: the state machine implemented

The three states are easy to draw in a diagram. What matters for real debugging is knowing exactly which variables drive the transitions and how to trace a breaker through a failure sequence. A breaker needs five pieces of state: a failure counter (or rolling window), a request counter, a threshold ratio, an open-timestamp, and the current state enum.

Worked numeric example. Parameters: failureThreshold = 0.5, minRequests = 10, windowMs = 60_000, cooldownMs = 30_000. Service B starts failing at T=0.

If instead the probe at T=43 fails, openedAt is reset to T=43, and the 30 s cool-down restarts — the next probe is allowed at T=73.

Pseudo-code implementing the transitions with comments on which line drives which row above:

// State variables (per breaker instance, per dependency)
state     = "CLOSED"
openedAt  = null
window    = []  // rolling array of {ts, success}

function call(fn, fallback) {
  if (state === "OPEN") {
    if (Date.now() - openedAt < COOLDOWN_MS) {
      return fallback()   // fail fast — rows T=14..T=42 above
    }
    state = "HALF_OPEN"    // row T=43
  }

  try {
    result = fn()         // actual dependency call
    record(success=true)
    if (state === "HALF_OPEN") {
      state  = "CLOSED"   // probe passed — row "Probe succeeds"
      window = []
    }
    return result
  } catch (err) {
    record(success=false)
    if (state === "HALF_OPEN") trip()  // probe failed — reset cool-down
    throw err
  }
}

function record(success) {
  now    = Date.now()
  window = window.filter(e => now - e.ts < WINDOW_MS)  // evict stale
  window.push({ts: now, success})

  if (window.length >= MIN_REQUESTS) {             // row T=7 — min-requests gate
    failures = window.filter(e => !e.success).length
    if (failures / window.length >= THRESHOLD) trip() // row T=13
  }
}

function trip() {
  state    = "OPEN"
  openedAt = Date.now()
}

How to debug & inspect it

A circuit breaker that has tripped is silent by default — requests fail fast with no call to the upstream, so the upstream logs show nothing. If you're seeing sudden 5xx/client errors without any corresponding upstream activity, the breaker is the first place to look.

The key Prometheus metrics to expose from a circuit breaker and alert on:

Tuning thresholds — the two failure modes:

• Threshold too sensitive (e.g. 20% with minRequests=5): the breaker trips on brief traffic spikes or normal startup noise, causing false-positive outages. Symptoms: breaker flips open and closed rapidly ("flapping"), logs show OPEN→HALF_OPEN→OPEN cycles within seconds.
• Threshold too permissive (e.g. 90%): the breaker doesn't trip until the dependency is almost completely down, so your service keeps hammering a failing upstream and the cascading failure isn't contained early enough.

Breaker tuning checklist:

1. Expose current state and failure rate as Prometheus metrics; set an alert for OPEN > 5 min.
2. Set a conservative minRequests (≥10) to avoid tripping on startup noise.
3. Verify every outbound call has both a connect timeout and a read timeout, and that timeouts are counted as failures by the breaker.
4. Configure a meaningful fallback — not just a bare error, but a degraded response or cached value.
5. Test the breaker in staging: inject failures (return 500 from a mock upstream) and verify the breaker trips at the expected failure count, stays open for the cool-down period, and closes on probe success.
6. Each dependency gets its own breaker instance — never share a breaker across two different downstream services.

By the numbers

Scenario: Service A calls a recommendation engine (Service B) at 200 req/s. Service B has a 5 s read timeout. A rolling window of 20 calls and a failure threshold of 50% are configured, with a 30 s cool-down.

Rolling window trace: when does the breaker trip?

Service B starts failing at T=0. The breaker window holds the last 20 calls. Each row below shows the window state after the indicated call:

Source for rolling-window mechanics: Martin Fowler — CircuitBreaker; production parameters from Azure Architecture Center — Circuit Breaker pattern.

Calls and resources saved by failing fast

Once OPEN at T=12, the breaker blocks all requests to Service B for the 30 s cool-down. At 200 req/s, how much waste is avoided?

In concrete terms: the breaker converts 30 000 thread-seconds of blocked capacity into near-zero thread-time (a fast fallback return costs microseconds). That is the quantified value of failing fast.

Decision math: threshold vs. flap trade-off

Choosing the failure threshold and minimum request count involves two opposing risks:

Break-even: the threshold should trip the breaker before thread-pool exhaustion. With a pool of P threads, an incoming rate of R req/s, and a timeout of T seconds, the pool fills in P / R seconds. The breaker must trip within that window. With P=200 threads, R=200 req/s, T=5 s: pool fills in 1 s, so the minimum-requests window must be ≤ R × 1 s = 200 calls. The trace above (20-call window, trips after ~12 s at 200 req/s) comfortably beats this threshold.

Sources & further reading

• Martin Fowler — CircuitBreaker
• Microsoft Azure Architecture — Circuit Breaker pattern
• AWS Builders' Library — Avoiding fallback in distributed systems
• Microsoft Azure Architecture — Bulkhead pattern