Design Case Studies · Lesson 10

Design: Video Conferencing API

A single "join meeting" click kicks off three separate sub-systems: a REST call that books a room, a WebSocket handshake that negotiates participants, and a UDP media stream that carries the actual video. The skill is keeping these planes separate and letting each one optimise for what it does best.

1 — Requirements

Before drawing a single endpoint, nail down what this system actually has to do. Video conferencing is a deceptively wide surface.

Functional requirements

• Room lifecycle: create a meeting, generate a shareable link, let others join, end it. A meeting has an id, a host, a title, and a scheduled start time.
• Participant signaling: every join/leave event must propagate to all current participants in real time — you cannot poll for this.
• Media routing: audio and video from each participant must reach every other participant with sub-200 ms end-to-end latency.
• Many participants: target 500 viewers, 25 active cameras. The naive approach of everyone sending to everyone (mesh) breaks down past 4–5 peers.
• Presence: who is muted, who is sharing screen, who has raised their hand — all participants need a consistent view of room state.

Non-functional requirements

• Latency: signaling under 100 ms; audio under 150 ms glass-to-glass; video under 200 ms.
• Reliability: temporary packet loss must not freeze frames — graceful degradation beats accuracy.
• Security: join tokens prevent uninvited guests; media must be encrypted in transit (SRTP/DTLS).
• Scalability: meeting creation is bursty; media load scales with participant count × bitrate, not meeting count.

2 — Design decisions

Decision 1: Separate the control plane from the media plane

This is the central insight. Two different jobs require two different protocols:

• The control plane manages rooms, issues tokens, and tracks who is in a meeting. It is infrequently used, needs a reliable delivery guarantee, and can tolerate 50–100 ms latency. REST over HTTP/2 is a natural fit.
• The media plane carries audio and video frames continuously. It needs sub-50 ms per-hop latency and must tolerate packet loss gracefully — late frames are discarded rather than retransmitted. This rules out TCP. UDP is the only realistic choice. (See the note on TCP vs UDP in Lesson 05 — Sockets.)

Decision 2: WebSocket for signaling

Between the REST control plane and the UDP media plane sits a signaling layer. Signaling messages are small (a few hundred bytes), must arrive in order, and must be pushed to all participants without polling. HTTP long-poll and SSE could handle server-to-client pushes, but participants also need to send to the server mid-call (mute, raise hand). WebSockets provide a persistent, bidirectional channel over TCP — exactly what signaling needs without any polling overhead.

Decision 3: Selective Forwarding Unit (SFU) instead of a mesh

In a peer-to-peer mesh, every participant sends their stream to every other participant. With N participants, each client uploads (N−1) streams. At N = 10, that is 9 simultaneous uploads — saturating a typical home connection. An SFU is a server that each client sends exactly one upstream to; the SFU then selects which streams to forward downstream to each subscriber. Upload cost per client stays fixed at 1× regardless of meeting size.

Decision 4: Short-lived join tokens

Meeting IDs are shareable links — they are not secrets. The join flow therefore requires a separate credential: a short-lived JWT that encodes {"meetingId", "participantId", "role", "exp"}. The client exchanges the meeting ID (plus auth) for a token from the REST API, then presents that token to the media server. The media server validates the token signature without calling back to the REST API — keeping the join hot path off the database.

Decision 5: Presence over the signaling WebSocket

Presence state (muted, hand raised, screen-sharing) is small, changes infrequently, and must be consistent across all participants. Routing it through the signaling WebSocket — not as media — keeps it decoupled from video packets and gives the server a single place to broadcast state changes.

3 — The API model

Control plane (REST)

These endpoints are called once per meeting lifecycle — not per frame.

# Create a meeting (host)
POST /v1/meetings
Authorization: Bearer <user-token>
Content-Type: application/json

{
  "title": "Weekly standup",
  "scheduled_at": "2025-09-01T09:00:00Z",
  "settings": { "max_participants": 25, "waiting_room": true }
}

# Response
HTTP/1.1 201 Created
{
  "id": "mtg_9kZxp2",
  "join_url": "https://meet.example.com/j/9kZxp2",
  "host_key": "hk_..."
}

# Join a meeting — returns a short-lived token + media server address
POST /v1/meetings/mtg_9kZxp2/join
Authorization: Bearer <user-token>

# Response: token is signed by REST API; media server validates it directly
HTTP/1.1 200 OK
{
  "participant_id": "pt_Uw83",
  "join_token": "eyJhbGc...",       // JWT, exp = 5 min
  "sfu_endpoint": "wss://sfu-eu-west.example.com",
  "signal_endpoint": "wss://sig-eu-west.example.com",
  "ice_servers": [                        // STUN/TURN for NAT traversal
    { "urls": "stun:stun.example.com:3478" },
    { "urls": "turn:turn.example.com:443", "credential": "..." }
  ]
}

Signaling plane (WebSocket)

After joining, the client opens a WebSocket to signal_endpoint and presents the join_token in the initial HTTP upgrade headers. All subsequent messages are JSON envelopes typed by "type".

// Client → server: announce presence and SDP offer
{
  "type": "participant.hello",
  "participant_id": "pt_Uw83",
  "display_name": "Divya",
  "sdp_offer": "v=0\r\no=..."   // WebRTC SDP for media negotiation
}

// Server → client: SDP answer from SFU
{
  "type": "sfu.answer",
  "sdp_answer": "v=0\r\no=..."
}

// Server → all: someone joined
{
  "type": "room.participant_joined",
  "participant": { "id": "pt_Uw83", "display_name": "Divya", "role": "attendee" }
}

// Client → server: presence update
{
  "type": "presence.update",
  "audio_muted": true,
  "hand_raised": false,
  "screen_sharing": false
}

4 — Evaluation & latency budget

Control vs media plane: why the separation holds

The REST control plane sees at most one request per participant per join — it scales with meeting creation events, not with call duration or packet rate. It can live behind a standard HTTP gateway with ordinary database-backed auth. The UDP media plane, by contrast, handles thousands of packets per second per participant (audio at 50 pps, 720p video at ~30 fps). Routing those through an HTTP layer would add at minimum one TCP round trip — 50 ms or more — per packet, which is incompatible with real-time audio perception. Lesson 05 (Sockets) covers why TCP's head-of-line blocking is lethal for media.

Latency budget for the join flow

SFU fan-out under load

At 25 cameras × 1.5 Mbps each, the SFU handles 37.5 Mbps of inbound media. If it forwards each stream to all 24 other participants: 24 × 25 × 1.5 = 900 Mbps outbound. In practice, SFUs use simulcast (clients send three resolution tiers) and subscriber-side bandwidth estimation to send each recipient only what their connection can absorb. A well-implemented SFU sends each participant at most its own subscribed bandwidth, collapsing the 900 Mbps theoretical figure to roughly the sum of each participant's available downlink.

Under the hood: the core mechanism

Three architectural facts determine everything about how a real SFU works. Understanding them lets you reason about scaling, latency, and failure modes without guessing.

Why mesh breaks: N(N-1) streams

In a pure peer-to-peer mesh each participant opens a direct connection to every other participant. With N active cameras, each client uploads N-1 streams and downloads N-1 streams:

The SFU collapses the upload cost to exactly 1 stream per sender regardless of participant count. Each client opens one UDP connection to the SFU and sends one stream; the SFU fans it out to all subscribers. Download cost per subscriber equals the number of streams they choose to receive — typically the N-1 other participants — but that load now falls on the SFU's outbound bandwidth, not the sender's upload.

Signaling (WebSocket/TCP) vs media (WebRTC/UDP): two completely separate pipes

These are not different message types on the same connection — they are different transport protocols with different network sockets:

A signaling message that delivers a "mute" event 80 ms late is acceptable. An audio packet delivered 80 ms late has already missed its playback slot — it is more disruptive than silence. These fundamentally different tolerance profiles demand separate transports.

Worked trace: 3 participants join in sequence

Alice is already in the room. Bob joins. Then Carol joins. Trace every stream that the SFU handles:

─── Alice joins (already in room) ───────────────────────────────
Alice → SFU:  1 upstream  (Alice's video)
SFU stores:   track[Alice] = stream_A

─── Bob joins ────────────────────────────────────────────────────
Bob  → REST API:  POST /v1/meetings/mtg_9kZxp2/join
REST → Bob:       { join_token, sfu_endpoint }

Bob  → Signaling: WS upgrade + participant.hello + sdp_offer
Signaling → SFU:  relay SDP offer
SFU → Signaling:  sdp_answer (SFU will accept Bob's upstream + subscribe to Alice)
Signaling → Bob:  sfu.answer

# ICE + DTLS handshake establishes Bob ↔ SFU UDP path
Bob → SFU:  1 upstream (Bob's video)       ← new stream in
SFU → Bob:  Alice's stream (forwarded)      ← 1 stream out to Bob
SFU → Alice: Bob's stream (forwarded)       ← 1 stream out to Alice

Signaling → Alice: room.participant_joined { id: Bob }  ← UI update

Total SFU state:  track[Alice], track[Bob]
Streams flowing: 2 upstreams in, 2 downstreams out (one per subscriber)

─── Carol joins ──────────────────────────────────────────────────
Carol → REST, Signaling, SFU:  same join flow

Carol → SFU:  1 upstream (Carol's video)
SFU → Carol:  Alice's stream + Bob's stream   ← 2 streams out to Carol
SFU → Alice:  Carol's stream                   ← Alice now gets 2 downstreams total
SFU → Bob:    Carol's stream                   ← Bob now gets 2 downstreams total

Signaling → Alice, Bob: room.participant_joined { id: Carol }

Final SFU state (3 participants, all cameras on):
  Upstreams:   3  (one per sender, fixed at O(1) per client)
  Downstreams: 6  (each participant receives 2 others' streams)
  Compare mesh: 6 upstreams + 6 downstreams = 12 connections, all from client machines

When simulcast is enabled, each sender actually sends 3 resolution tiers (e.g. 1080p, 360p, 180p). The SFU forwards only the tier that matches each subscriber's available bandwidth — determined by RTCP receiver reports. This collapses bandwidth further without the sender doing any extra work beyond the 3-tier upload.

Operating & debugging it

A video call failure looks different depending on which plane broke. The single most useful first step is identifying whether the symptom is in the signaling plane (WebSocket), the media plane (UDP/SRTP), or the join flow (REST + ICE). Each has distinct observable symptoms and distinct tools.

Production inspection

Symptom → cause → fix

// Client → signaling server
{ "type": "presence.update", "hand_raised": true }

// Signaling server → broadcast to all participants in room
{
  "type": "room.presence_changed",
  "participant_id": "pt_Uw83",
  "hand_raised": true
}

Sources & further reading

• WebRTC.org — Architecture overview
• MDN — Signaling and video calling
• BlogGeek.me — MCU vs SFU vs P2P: Which should you use?
• RFC 5245 — Interactive Connectivity Establishment (ICE)
• Zoom Engineering Blog — Optimizing Real-Time Communications
• RFC 3550 — RTP: A Transport Protocol for Real-Time Applications