Data & Formats · Lesson 01

Data representation & efficient communication

Your program holds a User object in RAM. The other program lives on a different continent. Before a single byte can travel the wire, that object must be flattened into a sequence of bytes — and faithfully rebuilt on the other end.

The root problem: memory addresses don't travel

Inside your process, a User object might live at memory address 0x7ff3a.... A pointer to it is just a number — meaningful only within that one running program, on that one machine, right now. Send the raw memory to another host and you'll get garbage: different OS, different language runtime, different byte ordering, and the address points nowhere useful.

Think of it like sharing a recipe. You can't hand someone a photocopy of your brain's neural patterns for "how to make lasagna." You write the recipe down in words (serialize), mail it (transmit), and the reader builds their own mental model of the dish from the description (deserialize). The written form is the agreed, portable representation.

Serialization is the act of converting an in-memory data structure into a portable sequence of bytes. Deserialization is the reverse. Together they form the round trip that makes distributed systems possible.

Schema-driven vs schema-less formats

Once you accept that data must be encoded, the next question is: who defines what the bytes mean? There are two broad camps.

A schema-less format (like JSON or XML) bundles the field names alongside the values. Every message is self-describing — you can read it without any external definition. The cost: field names are repeated in every single message, adding bytes, and nothing stops a sender from silently changing "email" to "emailAddress" without warning.

A schema-driven format (like Protocol Buffers or Avro) replaces field names with small integers defined in a shared schema file. The message itself carries almost no metadata — if you don't have the schema, the bytes are opaque. The payoff is a dramatic size reduction and guaranteed shape: both sides compile the same schema, so a mismatch is caught at build time, not at 3 AM in production.

Text vs binary: the quick take

Text formats encode everything as readable characters (UTF-8 bytes). They're debuggable with curl and a browser; they're also bigger, because the number 1234567 occupies 7 bytes as text vs 4 bytes as a 32-bit integer. Binary formats pack values into their native machine representations — smaller and faster to parse, but opaque to the naked eye.

For most public web APIs, JSON wins on developer experience. For high-throughput internal services where every microsecond and every kilobyte matters, binary formats earn their keep. Lesson 03 of this module goes deep on binary options.

Compression: trading CPU for bandwidth

Even with an efficient format, payload bytes can still be large. A hotel bill is verbose text — but it compresses beautifully because repeated patterns (dates, the hotel name, currency symbols) collapse down. HTTP lets you apply the same idea to API payloads.

gzip (RFC 1952) and Brotli are the two dominant HTTP compression algorithms. gzip is universal; Brotli (developed by Google) typically shrinks text 15–25% more than gzip at comparable CPU cost. On a 200 KB JSON payload, compression commonly yields a 5–10× reduction — slashing transfer time at the price of one compression and one decompression step. On fast internal networks where bandwidth is cheap and latency is low, compression may not be worth the CPU; on mobile or intercontinental links, it nearly always is.

Content negotiation: how client and server agree

HTTP has a built-in three-way handshake for agreeing on format and encoding. Think of it as a waiter taking a dietary order before cooking:

1. The client sends Accept: application/json (or application/xml, etc.) — "I can eat this."
2. The client also sends Accept-Encoding: gzip, br — "I can digest these compressions."
3. The server responds with Content-Type: application/json and Content-Encoding: gzip — "Here's JSON, gzip-compressed."

Content-Type describes the format of the body. Content-Encoding describes the transfer encoding (compression) layered on top. They are independent: you can have JSON compressed with Brotli, or plain-text XML with no compression.

# Client signals what it can accept
GET /api/v1/report HTTP/1.1
Host:        api.example.com
Accept:      application/json
Accept-Encoding: br, gzip;q=0.9

# Server responds with compressed JSON
HTTP/1.1 200 OK
Content-Type:     application/json; charset=utf-8
Content-Encoding: br
Vary:             Accept-Encoding

# <Brotli-compressed JSON bytes>

The Vary: Accept-Encoding header tells caches that two requests for the same URL with different Accept-Encoding values should be stored separately — otherwise a cache might serve a gzip blob to a client that only understands plain text.

The same object: JSON vs compact binary

To make the size difference concrete, here is a minimal user object serialized two ways. The JSON field names travel with every message; the binary version encodes field identity as small integers defined in a schema file.

// JSON — 98 bytes (pretty-printed for readability)
{
  "id":    9001,
  "name": "Priya Sharma",
  "role": "admin",
  "active": true
}
// Minified JSON — 57 bytes
// {"id":9001,"name":"Priya Sharma","role":"admin","active":true}

// Equivalent Protobuf binary — ~22 bytes (field numbers 1-4)
// 08 c9 46  12 0c 50 72 69 79 61 20 53 68 61 72 6d 61
// 1a 05 61 64 6d 69 6e  20 01
// Saving: ~60% vs minified JSON — grows with repetition

Under the hood: how it actually works

When your server returns a User object, it walks a four-step path before any bits leave the machine: the in-memory object is serialized to a JSON string, that string is encoded as UTF-8 bytes, those bytes are compressed, and the compressed bytes become the HTTP response body. Understanding each step explains why compression ratios vary and why small payloads sometimes bloat instead of shrink.

// The User object — same one from the binary comparison above
// {"id":9001,"name":"Priya Sharma","role":"admin","active":true}

// Stage 1 — Pretty-printed JSON string (UTF-8 bytes)
//   {
//     "id":    9001,
//     "name": "Priya Sharma",
//     "role": "admin",
//     "active": true
//   }
//   Size: 98 bytes

// Stage 2 — Minified JSON (whitespace removed)
// {"id":9001,"name":"Priya Sharma","role":"admin","active":true}
//   Size: 57 bytes  (-42% vs pretty-printed)

// Stage 3 — gzip-compressed (DEFLATE: LZ77 + Huffman)
//   Size: ~78 bytes  (+37% vs minified — header overhead dominates!)
//   gzip adds a 10-byte header + 8-byte trailer; payload too small to benefit

// Stage 4 — Brotli-compressed (LZ77 + Huffman + static dictionary)
//   Size: ~60 bytes  (~same as minified — dictionary helps a little)
//   Still not a win at 57 bytes; compression pays off above ~150–200 bytes

// On a real 50 KB JSON report body:
//   Raw JSON:   51 200 bytes
//   gzip:       ~9 200 bytes  (~82% reduction)
//   Brotli:     ~7 700 bytes  (~85% reduction)

How gzip works. gzip uses the DEFLATE algorithm, which chains two techniques. First, LZ77 scans a 32 KB sliding window of recently seen bytes and replaces any repeated sequence with a (distance, length) back-reference — the word "name" appearing ten times becomes one literal plus nine tiny pointers. Second, Huffman coding assigns shorter bit patterns to the most frequent symbols, squeezing further. The 10-byte gzip header and 8-byte CRC trailer mean payloads below roughly 150 bytes often grow after compression — the overhead bytes exceed the savings from any redundancy found in such a short string.

How Brotli works. Brotli is also LZ77 + Huffman, but it ships with a built-in static dictionary of roughly 13 000 common web strings — substrings like Content-Type, application/json, "status", "message", and hundreds of HTML/HTTP tokens. A back-reference into that dictionary costs just a few bits, so Brotli can compress web content it has never seen before by recognising the vocabulary. This is why Brotli beats gzip by 15–25% on typical API and HTML payloads at comparable CPU cost: it arrives already knowing most of the words in the document.

How to debug & inspect it

When compression is misbehaving — or you simply want to confirm it's working — curl gives you everything you need. The commands below let you measure actual wire sizes, inspect negotiation headers, and compare compressed vs uncompressed transfer times.

1. Check whether a Content-Encoding header is present in the response — use curl -v or the DevTools Network tab → Response Headers pane.
2. Verify that the server config has compression enabled and the correct MIME types listed (e.g., application/json must be in gzip_types; it is not always included by default).
3. Measure actual wire size with curl -o /dev/null -w "%{size_download}" and compare against the uncompressed figure to confirm real savings.
4. Confirm Vary: Accept-Encoding is set so CDN caches do not serve a Brotli-encoded response to a client that only supports gzip or plain text.
5. For Brotli: verify that the reverse proxy (nginx or Caddy) has the Brotli module compiled in — it is not bundled by default in many nginx distributions. Apache requires mod_brotli.

By the numbers

Compression turns payload size into a bandwidth and latency trade-off. The governing formula:

Scenario: a reporting API returning user analytics. Each JSON response is 12 KB uncompressed. After gzip it compresses to ~3 KB — a 75% reduction, typical for verbose JSON with repeated field names.

Payload size → compressed size trace at 5,000 req/s:

Worked trace — 12 KB JSON at 5,000 req/s:

Decision math — when to compress:

Sources: nginx gzip module — gzip_min_length; RFC 1952 — GZIP format specification; RFC 7932 — Brotli Compressed Data Format; web.dev — Enable text compression.

Sources & further reading

• MDN — Content-Encoding
• MDN — Content negotiation
• MDN — Vary header
• RFC 1952 — GZIP file format specification
• RFC 7231 §5.3 — Content negotiation in HTTP/1.1
• RFC 7932 — Brotli Compressed Data Format
• nginx gzip module documentation