Reliability & Scale · Lesson 11

Evolving APIs without breaking clients

Every live API is a promise made to unknown strangers. Keeping that promise while still shipping new features is one of the hardest acts of software craftsmanship — because the moment you break it, every caller breaks silently, at a time you didn't choose.

The tyranny of distributed callers

When you change a function inside one codebase, you find every call site with a compiler or a grep and fix them all in one commit. When you change a public API endpoint, your callers are out there in the wild — mobile apps that haven't updated, third-party integrations you've never heard of, scripts running in a bank's data center. You cannot find them all, and you cannot fix them. The only lever you have is backward compatibility: ensuring that old messages still work.

Think of the API like a train track gauge. The trains (clients) were built to fit the existing gauge. Widening the track is fine — old trains still run. Narrowing it, or moving the rails, derails everything already on the line.

The golden rule: additive changes are safe

An additive change adds something new without removing or modifying what already exists. It is the safest kind of evolution. A client that knows nothing about the new thing will simply ignore it — and that's the point. Concretely, these changes are safe to ship without a version bump:

• Adding an optional request field (the caller may omit it; old behavior is preserved).
• Adding a new response field (a well-behaved client ignores fields it doesn't recognise).
• Adding a new endpoint or a new HTTP method on an existing path.
• Adding a new enum value to a response (a client should treat unknown values as "unknown" rather than crash).
• Relaxing validation — accepting values you previously rejected (e.g. raising a max-length limit).

What actually breaks clients

A breaking change is anything that causes a previously working client to fail, misbehave, or need code changes. The canonical list:

The tolerant reader pattern

Jon Postel's robustness principle — "be conservative in what you send, be liberal in what you accept" — has a direct API design corollary called the tolerant reader. A client that applies this pattern:

1. Reads only the fields it needs; ignores everything else.
2. Treats unknown enum values as a known "unknown" sentinel rather than throwing.
3. Does not hard-code the exact set of keys in a JSON object.

If you build your clients this way, the producer gains freedom: adding fields to a response is truly free, because consumers won't blow up when they encounter something new. The danger is in the other direction — if consumers are fragile (strict schema validation that rejects unknown fields, for instance), every additive change by the producer can still break them.

Backward vs forward compatibility

Backward compatibility means new server code can handle old-format requests — clients that haven't updated still work. Forward compatibility means old server code can handle new-format requests — useful when you need to roll out a schema change to producers before consumers catch up. Both matter in practice: backward compatibility protects your existing consumers; forward compatibility protects rolling deployments where different server versions run simultaneously.

Expand-and-contract: the safe rename

Suppose you want to rename fullName to displayName in a response — a seemingly trivial change. Done naively it is a hard breaking change. The expand-and-contract pattern (also called parallel-change migration) turns one big breaking swap into three small safe steps:

Worked example: safe vs breaking change

Consider a user profile endpoint. Below are two scenarios — adding a field (safe) and renaming one (breaking without expand-and-contract).

# --- SCENARIO A: Safe additive change ---
# Old response
{
  "id": "usr_99",
  "fullName": "Laila Ahmadi",
  "email": "laila@example.com"
}

# New response — adds optional "avatarUrl"; old clients ignore it
{
  "id": "usr_99",
  "fullName": "Laila Ahmadi",
  "email": "laila@example.com",
  "avatarUrl": "https://cdn.example.com/avatars/99.png"  // NEW — additive
}
# ✓ Safe: tolerant readers ignore the new field.


# --- SCENARIO B: Breaking rename (BAD — do NOT do this) ---
# Renaming "fullName" → "displayName" in one deploy
{
  "id": "usr_99",
  "displayName": "Laila Ahmadi",   // RENAMED — old clients read undefined
  "email": "laila@example.com"
}
# ✗ Breaking: any client using response.fullName now gets undefined.


# --- SCENARIO C: Expand-and-contract (GOOD) ---
# Phase 1: Expand — emit BOTH fields
{
  "id": "usr_99",
  "fullName": "Laila Ahmadi",     // OLD — still present
  "displayName": "Laila Ahmadi",  // NEW — added; consumers start reading this
  "email": "laila@example.com"
}
# Phase 2: Migrate all consumers to read "displayName".
# Phase 3: Contract — remove "fullName" only once all consumers migrated.
# ✓ No consumer ever encounters a missing field.

Deprecation with the Sunset header

When you need to retire something — an endpoint, a field, an entire API version — the worst thing you can do is remove it silently. A Sunset header (standardised in RFC 8594) tells API consumers exactly when a resource will stop working, so their monitoring systems can surface the warning automatically.

# Response from a deprecated endpoint
HTTP/1.1 200 OK
Sunset: Sat, 31 Dec 2025 23:59:59 GMT
Deprecation: Tue, 01 Jul 2025 00:00:00 GMT
Link: <https://docs.example.com/migrate-v2>; rel="deprecation"
Content-Type: application/json

{
  // ... response body unchanged during deprecation period
}

Good deprecation also means: a changelog announcement with migration docs; enough runway for consumers (typically 6–12 months for public APIs); a migration guide linked from the Link header; and ideally a developer-portal banner. The header alone is not enough — but it's the machine-readable anchor that tooling and alert systems can act on.

Feature flags for safe rollout

Feature flags let you deploy new API behaviour to a small slice of consumers before everyone sees it. A common pattern for API evolution:

1. Deploy new behaviour behind a flag; it's off for all callers.
2. Enable it for internal consumers first — your own frontend, your own mobile app.
3. Enable for a beta cohort (opt-in partners, early adopters).
4. Ramp to 100% and remove the old code path only after monitoring shows no regressions.

For breaking changes you cannot avoid (a full major version), feature flags let you keep the new behaviour dark until all consumers have had time to migrate, rather than forcing a flag-day cutover.

Under the hood: expand-and-contract, step by step

The diagram earlier showed three phases as boxes on a timeline. This section makes each phase concrete — what code ships, what the wire looks like, how you detect that each phase is complete, and what tolerant-reader parsing looks like in practice.

The concrete scenario

The existing GET /v1/users/:id response contains fullName (a single string). The team wants to rename it displayName. There are three known server-side consumers (a web frontend, a mobile app, an internal reporting job) and an unknown number of third-party integrations. No version bump.

Phase 1 — Expand: dual-write both fields

Deploy a server change that emits both the old field and the new field, with identical values. Old consumers keep reading fullName and are unaffected. New consumers (including any you write today) can start reading displayName.

# Server response after Phase 1 deploy
{
  "id":          "usr_99",
  "fullName":    "Laila Ahmadi",     // OLD — still emitted
  "displayName": "Laila Ahmadi",  // NEW — added alongside
  "email":       "laila@example.com"
}

# Server-side pseudocode: write both from the same source field
def serialize_user(user):
    return {
        "id":          user.id,
        "fullName":    user.display_name,   # legacy alias
        "displayName": user.display_name,   # new canonical name
        "email":       user.email,
    }

This phase is safe to deploy at any time — no existing consumer breaks. The field fullName still has the same value, so code reading it still works. The deployment signal to advance to Phase 2: changelog published, all known consumer teams notified, and (ideally) a feature-flag or canary confirmed no regressions.

Phase 2 — Migrate readers: update each consumer to read the new field

Each consumer switches its read path from response.fullName to response.displayName. Both fields are still emitted, so this migration can be done one consumer at a time, in any order, with no coordinated downtime.

# Tolerant-reader parsing: the consumer only extracts what it needs
# and ignores everything else. This is what makes additive changes free.

# Python (Pydantic — extra fields ignored by default)
class UserResponse(BaseModel):
    id:          str
    displayName: str          # read new field
    email:       str
    # fullName not listed — Pydantic silently ignores it (model_config extra="ignore")

# Java (Jackson)
@JsonIgnoreProperties(ignoreUnknown = true)   // extra fields silently dropped
public class UserResponse {
    public String id;
    public String displayName;   // reads new field
    public String email;
    // fullName not declared — ignored
}

A non-tolerant client that has strict schema validation (e.g. Pydantic with extra="forbid", or a TypeScript type with a strict intersection) will reject a response that contains fullName because it was not in the expected shape. This is why Phase 1 is still safe when consumers are tolerant, but dangerous when they are strict. Audit your consumers' deserialization settings before Phase 1.

Signal to advance to Phase 3: access log analysis or server-side metrics confirm that no live traffic reads fullName. For mobile apps, this means the old app version (that reads fullName) has been sufficiently retired from your active user base. Instrument the server to log a warning whenever fullName is read — this makes the migration auditable.

Phase 3 — Contract: stop emitting the old field

Remove the fullName key from the server response. Any consumer that survived Phases 1 and 2 without switching is now broken — but you have empirical evidence (from Phase 2 monitoring) that none did.

# Server response after Phase 3 — clean
{
  "id":          "usr_99",
  "displayName": "Laila Ahmadi",
  "email":       "laila@example.com"
}

# Server-side: legacy alias removed
def serialize_user(user):
    return {
        "id":          user.id,
        "displayName": user.display_name,
        "email":       user.email,
    }

Phase timeline summary

How to debug & inspect it

Breaking changes almost always announce themselves through one of three failure modes: a consumer gets an unexpected undefined / null for a field they depend on; a consumer gets a 400 where they previously got 200 (validation tightened); or a consumer's data is silently wrong (semantic change — same field, different meaning). The debugging question is: which change broke which client, and when?

Detect a breaking change before you ship (contract tests)

The gold standard is consumer-driven contract testing (CDCT): each consumer publishes a "contract" describing exactly what fields and types it reads. A CI step on the producer side runs all consumer contracts against the new response shape and fails the build if any contract is violated. Tools: Pact (language-agnostic), Spring Cloud Contract.

Audit field usage in server access logs

If you lack CDCT, server-side request/response logging is the fallback. Instrument the serializer to track which response fields are accessed by which client (User-Agent or client ID). This is invasive in REST (the server cannot tell what the client does with a JSON response), but you can approximate it by tracking which clients are sending the old field in request bodies, or by observing behavior changes (error rates) after a deploy.

Debug checklist:

1. When a consumer reports breakage after a server deploy: check the git diff for the serializer — was a field removed or renamed in one step?
2. Run consumer-driven contract tests locally against the proposed change before merging.
3. For Phase 3 readiness: query access logs for any traffic reading the old field name; do not proceed until the count is zero (or acceptably low per your SLA).
4. Check the Deprecation and Sunset response headers on the deprecated field/endpoint — are they present and correctly dated?
5. For strict-schema consumer failures: confirm the consumer's JSON library is configured to ignore unknown properties.
6. Add a server-side metric/log line every time the old deprecated field is included in a response — set an alert on its traffic so you know when it reaches zero.

Sources & further reading

• Google API Design Guide — Compatibility
• RFC 8594 — The Sunset HTTP Header Field
• RFC 8594 — Deprecation header companion
• Martin Fowler — Tolerant Reader
• Martin Fowler — Parallel Change (expand-and-contract)
• Jon Postel — Transmission Control Protocol (RFC 761) §2.10 — robustness principle