Production at Scale · Walkthrough 04

Real-time chat fan-out at millions of connections (modeled)

Every open chat window is a persistent socket. When millions of users are connected simultaneously, the central problem is no longer throughput — it is connection state: where each user's socket lives, how many sockets one server can hold, and how a message crosses the boundary between servers to reach the right socket.

The core constraint: connection state is per-server memory

HTTP is stateless: any server in the fleet can answer any request, because no state is needed between calls. WebSocket connections break that assumption. Each persistent socket is a live piece of OS state — a file descriptor (fd), kernel send/receive buffers, and application-layer session data — that lives on exactly one server. You cannot route a message to a user unless you know which server holds their socket.

This is fundamentally different from scaling a REST API, where you add servers and distribute load freely. Scaling chat means tracking connection location and routing messages across server boundaries — the engineering cost that HTTP never had.

Per-connection cost: memory and file descriptors

Each WebSocket connection consumes three types of resource on the server:

How many connection servers? The arithmetic

Given the per-connection cost, the number of connection servers needed for a target concurrent connection count follows directly from available memory per server and the fd limit.

⚠️ Modeled, not measured — illustrative sizing

# Assumptions
mem_per_conn_kb       = 20           # ~20 KB total per connection (conservative)
server_ram_gb         = 32           # 32 GB RAM server, dedicate 28 GB to connections
usable_ram_kb         = 28 × 1_024 × 1_024 = 29_360_128 KB

# Connections per server (memory-limited)
conn_per_server_mem   = 29_360_128 / 20 ≈ 1_468_006  # ~1.4 M

# File descriptor limit (must also raise ulimit)
fd_limit              = 1_000_000   # tunable; some systems reach 10 M with epoll
conn_per_server_fd    = 1_000_000   # fd-limited at ~1 M per process

# Practical ceiling: min of memory and fd limits
conn_per_server       = min(1_468_006, 1_000_000) ≈ 1_000_000  # ~1 M

# Number of servers for 10 M concurrent connections
target_concurrent     = 10_000_000
servers_needed        = 10_000_000 / 1_000_000 = 10  # ~10 servers
# With 2× redundancy for failover: ~20 servers

# For 100 M concurrent connections
servers_100M          = 100_000_000 / 1_000_000 = 100  # ~100 servers (+ redundancy)

The result — roughly 10–100 connection servers for 10–100 million concurrent users — is a surprisingly small fleet. The challenge is not raw server count; it is the routing problem: how does a message originating on server 3 reach a recipient whose socket lives on server 17?

The routing problem: crossing server boundaries

User A (on server 3) sends a message to user B (on server 17). Server 3 receives the WebSocket frame, but B's socket is not on server 3. How does the message get there?

The naive approach — server 3 directly calls server 17 over HTTP — requires every connection server to know about every other connection server, and to maintain a routing table of which user is on which server. With 10–100 servers this is manageable, but the table must be kept consistent as users connect, disconnect, and reconnect. A centralised pub/sub bus solves this more cleanly.

Pub/sub bus design: channels, subscriptions, and delivery guarantees

Each connection server subscribes to one channel per room (or per user) that has an active member on it. When a message arrives on a channel, the bus delivers it to all subscribed servers; each server then fans the message out to the local sockets in that room.

Presence and message ordering

Presence is the signal "user X is online." Tracking it requires a shared store updated on connect and disconnect. A Redis Hash keyed by user_id with a value of {server_id, last_heartbeat_ts} works well: the connection server writes on connect, deletes on clean disconnect, and refreshes the heartbeat every 30 seconds. A separate sweeper removes stale entries (heartbeat > 90 s old) to handle unclean disconnections.

Message ordering is harder. The pub/sub bus delivers messages to multiple servers and the order of delivery is not guaranteed to match insertion order. Each message should carry a monotonically increasing sequence id scoped to the room, generated at the point of acceptance (before publishing to the bus). The server that receives the send request assigns the id atomically (e.g., Redis INCR on a per-room counter). Clients order their local message list by sequence id, not by arrival time. Gaps in the sequence trigger a client-side "fetch missed messages" request against the message store.

# Message acceptance flow (server that receives the client's send)
def accept_message(room_id, sender_id, body):
    seq_id = redis.incr(f"seq:{room_id}")         # atomic, monotonic
    msg = {
        "id":      generate_uuid(),
        "seq":     seq_id,
        "room":    room_id,
        "sender":  sender_id,
        "body":    body,
        "ts":      utcnow_ms(),
    }
    db.insert("messages", msg)                  # durable store first
    redis.publish(f"room:{room_id}", json(msg)) # then fan-out
    return {"seq": seq_id, "status": "sent"}

# Client side: detect and fill gaps
if received_msg.seq > local_last_seq + 1:
    fetch_missed(room_id, since_seq=local_last_seq)  # backfill from REST endpoint

Worked: group message fan-out for a 500-member room

A message is sent in a room with 500 members. Derive the full fan-out cost from message acceptance to final socket write.

⚠️ Modeled, not measured — group fan-out arithmetic

# Setup
group_size           = 500      # members in the room
connection_servers   = 10       # total connection servers
members_per_server   = 500 / 10 = 50   # assume even distribution

# Step 1: one inbound WebSocket frame from sender
inbound_frames       = 1

# Step 2: accepting server publishes to bus (1 write)
bus_publishes        = 1

# Step 3: bus delivers to all subscribed servers
#          (all 10 servers subscribe to room:42)
bus_deliveries       = 10

# Step 4: each server writes to its local room members' sockets
#          server A: 50 members - 1 sender = 49 writes
#          servers B–J: 50 writes each
socket_writes_total  = 49 + (9 × 50) = 499   # = group_size - 1

# Fan-out ratio
fan_out_ratio        = 499 / 1 = 499×

# Throughput: 1,000 messages/sec to a 500-person group
# requires: 1,000 × 499 = 499,000 socket writes/sec across the fleet
fleet_socket_writes  = 1_000 × 499 = 499_000 writes/sec
per_server           = 499_000 / 10 = 49_900 writes/sec   # very achievable

# Large group (5,000 members): same message rate costs 5,000× more
large_group_writes   = 1_000 × 4_999 = 4_999_000 writes/sec
per_server_large     = 4_999_000 / 10 = 499_900 writes/sec   # heavier

Offline delivery and message persistence

A user who is offline when a message is sent has no socket — there is nothing to write to. The message must be stored durably and delivered when the user reconnects. This is architecturally separate from the real-time path:

1. Write to durable message store before publishing to the bus (as shown in the pseudocode above). The store (e.g. Cassandra, PostgreSQL, or a purpose-built message store) is the source of truth.
2. Record an unread indicator per (user, room) in a fast counter store — a Redis Hash or sorted set keyed by user id is common. This tells the client how many missed messages to fetch on reconnect.
3. On reconnect: the client sends its last-seen sequence id; the server fetches messages with seq > last_seen from the durable store and delivers them before joining the live pub/sub stream.

File descriptor and OS tuning: the constraints in detail

Each TCP connection (including WebSockets) consumes one file descriptor on the server. The Linux defaults are conservative and must be raised for connection servers holding hundreds of thousands of sockets. See the sockets fundamentals lesson for the underlying OS model.

Beyond file descriptors, the other OS resource that constrains connection count is TCP buffer memory. Each socket has kernel send and receive buffers; their defaults are set by net.core.rmem_default and net.core.wmem_default. Reducing buffers (e.g. to 4 KB each) doubles the connection density at the cost of reduced burst throughput per connection — a good trade for chat, where individual messages are small.

Sources & further reading

• Discord Engineering Blog — scaling concurrent connections — public discussion of connection architecture and per-connection resource costs.
• Facebook Engineering — Building Facebook Messenger (2011) — early public account of message routing, delivery, and presence at scale.
• Slack Engineering — Real-Time Messaging — Slack's public description of their connection and pub/sub architecture.
• Redis Pub/Sub documentation — channel subscription model, delivery semantics, and throughput characteristics.
• WebSockets & real-time (this course) — the WebSocket upgrade handshake, frame format, and browser model.
• Sockets fundamentals (this course) — file descriptors, TCP buffer sizing, and OS-level connection limits.
• Event-driven pub/sub (this course) — bus semantics, topic design, and delivery guarantees.
• Queue & backpressure simulation (this course) — interactive pub/sub throughput exploration.
• Design Case Study: Messenger (this course) — full end-to-end system design for a chat application.