Data & Formats · Lesson 03

Binary formats: Protocol Buffers & Avro

When a system processes millions of messages per second, the overhead of printing and parsing field names in JSON stops being a convenience and starts being a bottleneck. Binary formats eliminate that overhead — but they demand a different way of thinking about schemas.

Why binary: the tax on human readability

Every JSON message carries a metadata tax you never think about: the field names. In {"latitude": 37.7749, "longitude": -122.4194}, the values are 15 characters; the field names are 21. For a single response it doesn't matter. For a sensor that broadcasts its position 500 times per second to 10 000 subscribers, those 21 characters per message add up to over 100 MB/min of pure overhead — bytes that carry zero information once the schema is agreed.

Binary formats eliminate the tax by replacing field names with small integers. Both sides agree on a schema — a shared document that says "field number 1 is latitude, field number 2 is longitude" — and the wire carries only the numbers, not the names. A three-byte field tag instead of a ten-byte string; a four-byte float instead of its ten-character text representation.

The trade-off is direct: binary messages are not human-readable. You cannot paste them into a terminal and understand them. You always need the schema, and usually a code generator, to work with them. For public developer APIs where debugging experience matters, that cost is often too high. For internal, high-throughput infrastructure — event pipelines, gRPC microservices, mobile push — it's the right call.

Protocol Buffers: field numbers as the contract

Protocol Buffers (Protobuf), developed at Google, centres everything on a .proto file. This schema assigns each field a permanent integer — the field number. The generated code turns that file into strongly-typed classes in Go, Java, Python, etc. The wire format carries field numbers, never names.

// location.proto — a minimal Protobuf schema
syntax = "proto3";
package telemetry;

message Location {
  double latitude  = 1;   // field number 1 — permanent
  double longitude = 2;   // field number 2 — permanent
  int64  timestamp = 3;   // Unix ms; field number 3
}

// Safe additions — new fields get new numbers
message LocationV2 {
  double latitude  = 1;
  double longitude = 2;
  int64  timestamp = 3;
  float  altitude  = 4;   // added in v2 — old consumers ignore it
  string sensor_id = 5;   // added in v2 — old consumers ignore it
}

The code generator (invoked as protoc) reads this file and emits serialization/deserialization code for your target language. Serializing a Location in Go is a method call; the developer never writes a loop over bytes.

Avro: schema stored alongside data

Apache Avro takes a different angle: the schema is written in JSON and is often stored alongside or embedded in the data file. In the Kafka ecosystem, Avro schemas are registered in a Schema Registry — a shared catalogue that assigns each schema an integer ID. Messages carry only that small ID; consumers look up the full schema from the registry before decoding.

Avro's headline feature is schema resolution: when a producer and consumer have slightly different versions of a schema, Avro can map old fields to new fields by name — useful in event-sourcing or data lake scenarios where you can't atomically redeploy all producers and consumers at the same time.

// Avro schema (JSON notation)
{
  "type":   "record",
  "name":   "Location",
  "fields": [
    { "name": "latitude",  "type": "double" },
    { "name": "longitude", "type": "double" },
    { "name": "timestamp", "type": "long"   }
  ]
}

// Adding a field safely: supply a default so old consumers
// can decode records that lack the new field
{ "name": "altitude", "type": ["null", "float"], "default": null }

When binary beats JSON: the gRPC connection

Google's gRPC framework pairs HTTP/2 with Protobuf. The combination gives you multiplexed streams over a single TCP connection, strongly-typed IDL-generated clients, bidirectional streaming, and binary payloads — all in one package. gRPC is the standard wire protocol for internal microservice communication at companies operating at Google / Netflix scale.

The rule of thumb: choose binary when all three of the following are true — (1) both endpoints are internal services you control; (2) throughput or latency is genuinely constrained; (3) you have a CI pipeline that can compile and distribute schema changes. If any condition is false, lean JSON until proven otherwise.

Under the hood: how it actually works

Every field on the wire starts with a tag byte computed by the formula tag = (field_number << 3) | wire_type. Wire types are small integers: 0 = varint, 1 = 64-bit fixed, 2 = length-delimited (string / bytes / embedded message), 5 = 32-bit fixed. So field 1 carrying a varint has tag byte 0x08 = (1 << 3) | 0. Field 2 carrying a varint has tag 0x10 = (2 << 3) | 0. Field 3 carrying a length-delimited string has tag 0x1a = (3 << 3) | 2. The decoder reads the tag, extracts the field number and wire type, and knows exactly how to read the bytes that follow — without needing the field name at all.

Values themselves use varint encoding for integers: small numbers fit in fewer bytes. Values 0–127 fit in a single byte. Values 128–16383 fit in two bytes. The most-significant bit (MSB) of each byte is a continuation bit: 1 means more bytes follow; 0 means this is the final byte of the integer. The integer 1 therefore encodes as 0x01 (MSB is 0 — done). The integer 300 encodes as 0xac 0x02: first byte 0xac has MSB 1 (more follows) and carries the low 7 bits; second byte 0x02 has MSB 0 and carries the remaining high bits.

// .proto definition
message User {
  int32  id   = 1;    // field 1, wire type 0 (varint)
  string name = 2;    // field 2, wire type 2 (length-delimited)
}

// Encoding User { id: 42, name: "Ada" }
// Field 1 (id=42):
//   tag = (1 << 3) | 0 = 0x08
//   value 42 = 0x2a (varint, fits in 1 byte, MSB=0)
//   bytes: 08 2a

// Field 2 (name="Ada"):
//   tag = (2 << 3) | 2 = 0x12
//   length = 3 (0x03)
//   UTF-8 bytes of "Ada" = 41 64 61
//   bytes: 12 03 41 64 61

// Full message: 7 bytes total
08 2a  12 03 41 64 61
// ^tag ^42  ^tag  ^len ^"Ada"

// JSON equivalent: {"id":42,"name":"Ada"} = 20 bytes (minified)
// Binary is 65% smaller for this tiny message

The compactness advantage is structural, not incidental. In JSON, the key "id" costs 4 bytes (quotes + letters + colon), and "name" costs 7 bytes — 11 bytes of overhead just for two field names in a 20-byte message. Protobuf uses 2 tag bytes total for those same two fields. Scale that up: a message with 10 fields and average key length 8 bytes carries roughly 80 bytes of JSON key-plus-colon overhead, replaced by roughly 10 tag bytes in Protobuf. Before even accounting for the more efficient numeric encoding (4-byte float vs. its 12-character text form), a typical API message is already 3–5× smaller on the wire.

How to debug & inspect it

Because Protobuf is binary and opaque to the naked eye, debugging requires dedicated tooling. The protoc --decode_raw command decodes any Protobuf binary blob without the schema — it shows field numbers and wire types, but not field names. When you do have the .proto file, protoc --decode=package.MessageName produces a full human-readable text representation with field names. For gRPC specifically, grpcurl lets you call a live endpoint and read the JSON-decoded response without writing any client code.

For web-based inspection without installing local tools, use protoscope (a Protobuf hex dumper available as a Go binary) or the online Protobuf Decoder — paste hex bytes and it returns a field-by-field breakdown with wire types.

Debug checklist:

1. Use protoc --decode_raw on a captured binary blob to confirm the field numbers match your .proto definition before assuming a parsing bug.
2. When adding a field, check the .proto for reserved declarations — if the number you want is reserved, pick the next available number.
3. Run protoc --lint (or buf lint) in CI to catch naming and numbering violations before they reach production.
4. For Avro: after adding a field, test schema compatibility with avro-tools or the Confluent Schema Registry compatibility check API before deploying the new producer.
5. Keep a changelog of field number allocations in the .proto file as comments — future engineers need to know which numbers are reserved and why.

message Order {
  int64  order_id     = 1;
  string customer_id  = 2;
  double total_cents  = 3;  // BUG: should be int64
}

message Order {
  int64  order_id      = 1;
  string customer_id   = 2;
  // Field 3 (total_cents double) is REMOVED — mark reserved
  reserved               3;
  reserved               "total_cents";  // prevents name reuse
  int64  total_cents_v2 = 4;  // correct type, new number
  string currency_code  = 5;  // new field — old consumers ignore
}

Sources & further reading

• protobuf.dev — Protocol Buffers overview
• protobuf.dev — Language guide (proto3)
• Apache Avro — Specification
• gRPC — Introduction
• Martin Kleppmann — Schema evolution in Avro, Protocol Buffers and Thrift
• protobuf.dev — Encoding (wire format)
• grpcurl — command-line tool for gRPC