Reliability & Scale · Lesson 14

Scaling from 1 to 1,000,000 users

The architecture graveyard is full of systems that worked perfectly at launch. They weren't killed by bad code — they were killed by the architecture that was right at Stage 1 still running the show at Stage 5.

Why scale kills systems

The most important insight in this lesson is also the most counter-intuitive: you do not design for one million users on day one. You design a system that can evolve through scale stages without requiring a full rewrite each time. The startup that builds a CQRS + event-sourced + sharded architecture for its first 200 users isn't being forward-thinking — it's writing a system nobody on the team fully understands, that takes three times as long to build, and that has more failure modes than the problem it was built to handle.

Instead, the bottleneck shifts as you grow, and that shift is predictable:

1. First, you hit CPU limits on the single app server — too many requests, not enough compute.
2. Then, the database becomes the bottleneck — it's the one component that every app server talks to, and it can't scale by simply adding machines the way app servers can.
3. Then, the write path saturates — reads can be spread across replicas, but all writes still go through one primary.
4. Finally, geographic latency dominates — the speed of light becomes your new bottleneck, and no amount of compute fixes a 180 ms round-trip to a data centre on the other side of the planet.

Each of these shifts demands a different architectural response. The skill is recognising which shift you're in — using real metrics, not guesswork — and applying the right technique at the right time.

Vertical vs horizontal scaling

Every scaling decision starts with a fundamental choice: make the machine you have bigger, or add more machines.

Vertical scaling (scale up)

Buy a server with more CPU cores, more RAM, or faster storage. The application needs zero code changes — it simply has more resources. This is the right first move; it's cheap, fast, and introduces no new failure modes. The problem is the ceiling. At some point you run out of machine to buy. AWS's largest general-purpose instance (as of 2025) has 448 vCPUs and 6 TB of RAM. That is genuinely enormous — most systems never need it — but it is a hard ceiling, and the cost curve steepens sharply before you get there.

Horizontal scaling (scale out)

Add more machines and spread the load across them. Instead of one waiter running faster to serve a full restaurant, you hire ten waiters. Each one handles a fraction of the tables, and you can hire more on a busy Friday night and let some go on a slow Tuesday.

Horizontal scaling can extend almost without limit. Cloud providers have enough physical capacity that you will almost certainly exhaust your budget before you exhaust the pool of available instances. The price is architectural: horizontal scaling only works cleanly when each server carries no local state. A server that holds per-user session data in its own memory is "sticky" — users must be routed back to the same server on every request. If that server fails, all of its users lose their sessions. And every new server you add is a potential destination that lacks the session data for half your users.

The enabling move is to externalise state — sessions, caches, uploaded files — to a shared store that every server can reach. Once you do that, every app server becomes interchangeable, and the load balancer is free to send any request to any server. This connects directly to the session-affinity problem covered in the load-balancing lesson and the caching layer discussed in the caching lesson.

The architecture progression: seven stages

What follows is a map of the seven stages most web systems pass through on their way from launch to serious scale. Each stage describes what the system looks like, what eventually breaks, and what you add to fix it. The numbers are approximate — your real thresholds depend on request complexity, DB schema, and read/write ratio. Use capacity estimation to know when you are approaching the next boundary.

Stage 1 — 0 to 1,000 users: the single box

Everything lives on one server: the application process, the database, and static file serving. A $20–40/month VPS handles this with room to spare for most CRUD applications. The deploy story is git push and systemctl restart app. The monitoring story is one SSH session.

What breaks: any component failure means a total outage. The app process and DB compete for the same CPU and RAM, so a spike in query load starves the app, and vice versa.

Correct response: nothing. Ship fast, validate your product, and collect real traffic data. Every premature architectural addition at this stage is debt you carry for months before it pays off.

Stage 2 — 1,000 to 10,000 users: split the database

Move the database to its own server. The app server and DB server each get dedicated CPU and RAM. The app can now spike without taking the DB down, and the DB can run autovacuum and index maintenance without starving HTTP handlers. This is also the moment to add automated backups and a basic alerting stack.

What breaks: the app server is still a single point of failure. A deployment takes the whole system down. The DB is starting to feel read pressure from a growing number of users hitting frequently-read rows.

What you add: a separate DB server (the two can talk over a private network interface). Automated nightly backups. A monitoring agent (Datadog, Prometheus + Grafana, or even UptimeRobot for the basics).

Stage 3 — 10,000 to 100,000 users: add a cache

A pattern emerges in the query log: roughly 80% of DB reads are for the same hot rows — user profiles, configuration tables, popular feed items, product catalogue entries. Every one of those reads hits the DB, waits for a disk or buffer-pool lookup, and returns the same bytes it returned a millisecond ago for a different user. A cache absorbs this flood by storing those hot bytes in RAM, where they can be returned in under a millisecond without touching the DB at all. See the caching lesson for the full cache invalidation playbook.

What breaks: write QPS is growing, and the cache adds invalidation complexity — stale reads become a risk. App server CPU begins to peak during traffic spikes because it's handling both web requests and cache-miss DB calls in the same process.

What you add: Redis or Memcached in front of the DB for hot reads. A CDN (Cloudflare, CloudFront, Fastly) for static assets — this alone removes a surprising fraction of origin requests.

Stage 4 — 100,000 users: load balancer + stateless app servers

One app server is now your ceiling for compute and your only failure mode for uptime. The fix is to run multiple app servers behind a load balancer — but this only works if the servers are genuinely stateless. Move session state to Redis (the same instance you likely added in Stage 3). Now the load balancer can route any request to any app server, and you can add more servers without users noticing. See the load-balancing lesson for algorithm choices.

What breaks: the DB is now the single funnel for all data operations. Five stateless app servers can handle five times the request volume, but all of that volume's reads and writes still flow through a single database primary. The DB becomes the new ceiling.

What you add: a load balancer (AWS ALB, NGINX, HAProxy). At least two stateless app servers. Session state moved to Redis. Deployment becomes rolling: take one server out, deploy, put it back, repeat — zero downtime.

Stage 5 — 500,000 users: read replicas

At a typical read:write ratio of 10:1, roughly 90% of the DB queries are reads. Adding 1–3 read replicas — databases that receive a continuous stream of write operations from the primary via replication and can serve read queries independently — lets you distribute 90% of the DB load across multiple machines. Direct all SELECT queries to the replicas; keep all INSERT, UPDATE, and DELETE on the primary. With three replicas, you've effectively multiplied your read capacity by 4× while the primary handles only writes.

What breaks: replication lag. Replicas receive the primary's write stream asynchronously, which means they may be 10–100 ms behind. A user who writes something and immediately reads it back may get stale data if the read hits a replica before the write has propagated. Write QPS is still limited to the single primary — you've solved the read problem but not the write problem.

What you add: read replicas (Postgres streaming replication; MySQL binlog replication). Query routing logic in the application or a proxy (PgBouncer, ProxySQL): writes go to primary host, reads go to replica host(s). Monitor replication lag as a first-class metric — alert at >500 ms.

Stage 6 — 1,000,000+ users: shard writes + async queues

Read replicas cannot help write throughput — all writes still hit one primary. When the primary can no longer absorb write QPS, two approaches run in parallel. First, DB sharding: partition your data across multiple independent primary databases using a shard key (commonly user_id % N → shard N). Each shard handles a fraction of the writes. Second, async queues: identify writes that don't need to be synchronous — sending email, recalculating feed rankings, generating thumbnails — and move them off the request path into a message queue (SQS, Kafka, RabbitMQ). A fleet of background workers drains the queue independently. See the event-driven lesson for queue mechanics.

What breaks: sharding introduces cross-shard complexity. A query that previously joined two tables in one database now potentially joins across shards, which may require an application-level scatter-gather. Analytics queries become expensive. Async queues introduce eventual consistency — callers must accept that some operations complete seconds or minutes after the API call returns.

What you add: horizontal DB sharding, or migrate to a NewSQL database (CockroachDB, PlanetScale, Spanner) that handles sharding transparently. A message queue for async work. Worker processes that consume the queue.

Stage 7 — multi-region

Beyond raw throughput, geographic latency becomes the ceiling. A user in Singapore talking to a data centre in Virginia experiences 180–220 ms of round-trip time before a single byte of application logic runs. The fix is to run a full application stack in each major region and route users to the nearest one via DNS geo-routing (AWS Route 53 latency routing, Cloudflare, etc.). This is the hardest distributed systems problem: how do you keep databases in different continents consistent with each other? CAP theorem guarantees you must trade either consistency or availability when a network partition separates the regions. Most systems choose to write globally and read locally, or use active-active with conflict resolution — a decision that depends deeply on the application's consistency requirements.

What breaks: cross-region writes, split-brain scenarios during network partitions, data sovereignty requirements (GDPR mandates that EU user data stays in the EU), and operational complexity that multiplies with every region you add. This territory is covered in detail in the high-availability lesson.

What you add: DNS-based geo-routing. Per-region app stacks. A global DB layer or a replication strategy (read-local, write-global; or active-active with a CRDT-based conflict resolution strategy).

Under the hood: why stateless enables horizontal scale

The word "stateless" appears constantly in scaling discussions. Here is exactly what it means mechanically, and why it matters so much.

The stateful problem in detail

A typical stateful app server stores session data in an in-process data structure — a HashMap<userId, SessionData> in the JVM heap, a Python dict in the process's memory, or a module-level variable in a Node.js process. When user 42 logs in, that session object is created and lives in Server A's RAM. The session might look like:

// Server A's in-memory session store (simplified)
sessions = {
  "tok_7f3a9c": {
    userId:    42,
    email:     "ali@example.com",
    role:      "admin",
    loggedInAt:1718000000
  }
}

Request 2 from user 42 arrives. The load balancer uses round-robin and sends it to Server B. Server B has no tok_7f3a9c key in its session map — it was never told about the login. Server B either returns a 401 (user appears logged out) or panics trying to look up a null session.

The "fix" teams reach for is sticky sessions: configure the load balancer to always send user 42 to Server A. This works until Server A crashes or is redeployed. Then all of Server A's sessions are gone, and every user it was serving gets logged out simultaneously. You've also negated the load balancer's ability to actually balance — if half your user base happens to be sticky to one server that's now running a slow query, the other servers sit underutilised.

The stateless solution

A stateless server stores zero per-user data in local memory. The session object lives in Redis — a shared, external store that every app server can read and write. The flow becomes:

1. User 42 logs in. Server A writes the session to Redis under key sess:tok_7f3a9c. Server A returns the token in a cookie. Server A's own memory holds nothing.
2. User 42 makes a second request. The load balancer sends it to Server B (round-robin). Server B reads the cookie, extracts tok_7f3a9c, and fetches sess:tok_7f3a9c from Redis in <1 ms. Server B has full session context. User 42 sees no difference.
3. Server A crashes. Requests that would have gone to A are rerouted to B and C. Redis still has all session data. No users are logged out.
4. You add Server D. The load balancer routes ~25% of traffic to it. Server D reads sessions from Redis like everyone else. You added a machine with zero application changes.

Worked numeric trace

This is what the difference looks like in concrete numbers at 1,000 requests per second:

The key phrase: statefulness is a scaling ceiling; statelessness is a scaling floor.

Finding and fixing the bottleneck

The cardinal sin in scaling work is adding more of the wrong thing. Adding app servers when the bottleneck is the DB costs money and complexity without improving performance. Adding read replicas when the bottleneck is app-server CPU costs money and introduces replication lag for no gain. The only correct order is: measure first, then fix the component that is actually saturated.

Sources & further reading

• Google SRE Book — Site Reliability Engineering
• AWS Well-Architected Framework
• Werner Vogels — A Word on Scalability
• The Twelve-Factor App — VI. Processes (stateless)
• High Scalability — architecture case studies