Design Case Studies · Lesson 20

Design: IoT Data Ingestion API

IoT networks feature millions of resource-constrained devices streaming high-frequency sensor readings. The primary bottleneck is the ingestion path: handling millions of write requests per second without saturating server network interfaces or overwhelming the database. We must design a highly scalable, append-only ingestion pipeline using binary protocols, pre-aggregated batches, and optimized time-series storage.

Requirements

Designing an IoT ingestion API requires balancing device battery, network bandwidth, and server IOPS:

• Ingestion Scale. Support $1,000,000$ active devices transmitting sensor logs.
• Write Path Durability. Ingested data must be reliably committed to disk; however, minor telemetry loss (e.g., 0.01% of temp readings) is acceptable if it prevents complete pipeline lockup.
• Bandwidth Constraints. Devices operate on cellular or satellite links (pay-per-byte billing); payloads must be as small as possible.
• Database Efficiency. Relational index updates on every insert ($O(\log N)$ b-tree operations) will fail at $100,000+$ writes/sec. The database must support append-only high-rate writes.

Design decisions

Transport Protocols: MQTT vs. CoAP vs. HTTP/2 Batch

For high-frequency, resource-constrained telemetry, standard HTTP/1.1 REST is highly inefficient due to TCP handshake and text header overhead (around 500+ bytes per request). The primary options are:

• MQTT (Message Queuing Telemetry Transport). Runs over TCP. Lightweight publish/subscribe model. Best when devices maintain persistent connections and require bidirectional command-and-control messaging.
• CoAP (Constrained Application Protocol). Runs over UDP. Mimics REST semantics but uses a tiny binary header. Ideal for extremely battery-constrained or memory-constrained microcontrollers.
• HTTP/2 or HTTP/3 REST with Batching. Runs over multiplexed TCP/QUIC. Ideal for gateways or smarter edge devices that can buffer readings locally and upload them in batches (e.g. once every 60 seconds) over TLS.

Data Serialization: JSON vs. Protobuf

To reduce payloads, we avoid verbose JSON text. A typical JSON telemetry payload consumes ~150 bytes. A corresponding Protobuf (Protocol Buffers) payload compiles to just ~24 bytes (an 84% reduction), drastically lowering client transmission energy and data cost.

Time-Series Storage Architecture

Traditional relational tables index rows using B-Trees, which requires updating the index file on disk for every insert. Under high write loads, this causes random disk IOPS exhaustion. We use a Time-Series Database (TSDB) or TimescaleDB hypertables. Hypertables partition data into time-based chunks (e.g., 1-day tables). Since new data always targets the current active partition, the indices fit entirely in RAM, allowing constant-time ($O(1)$) append operations.

The API model

Device Batch Telemetry API (HTTP/2 REST)

For edge gateways and aggregated uploads, we expose a batch endpoint accepting a compressed binary payload containing a sequence of sensor events.

# Upload batched binary Protobuf payload
POST /v1/telemetry/upload HTTP/2
Host: ingest.example.com
Content-Type: application/x-protobuf
Content-Encoding: gzip
X-Device-Signature: hmac-signature-here

[Compressed Protobuf Binary Data]

# Success response
HTTP/2 202 Accepted
Content-Type: application/json

{
  "status": "queued",
  "records_processed": 120
}

Protobuf Payload Definition

This compact schema defines the batch request. By using delta-compression for timestamps, we reduce payload sizes even further.

syntax = "proto3";

message MetricReading {
  uint32 metric_id = 1;      // ID mapping (e.g., 1=temp, 2=humidity)
  double value = 2;          // Numeric sensor value
  sint32 time_offset_ms = 3; // MS delta relative to batch base_timestamp
}

message TelemetryBatch {
  string device_id = 1;
  uint64 base_timestamp_ms = 2; // Absolute Epoch timestamp for reference
  repeated MetricReading readings = 3;
}

Under the hood: Ingestion pipeline

To handle high-throughput writes without database locks, the ingestion path isolates the HTTP/MQTT listeners from the database using a durable message broker.

By the numbers: telemetry scaling & storage math

Let's calculate the ingestion load and storage footprint for a large-scale telemetry platform.

Governing Equations

• Total Raw Inbound Event Rate: $$Event_{rate} = \frac{N_{devices} \times F_{readings}}{T_{interval}}$$ Where $N_{devices}$ is the number of devices, $F_{readings}$ is the readings per device, and $T_{interval}$ is the interval in seconds.
• Daily Raw Data Size (Uncompressed): $$Size_{raw} = Event_{rate} \times Payload_{bytes} \times 86,400\text{ sec/day}$$
• Double-Delta TSDB Compression: High-density time-series data compresses exceptionally well using Gorilla compression (delta-of-delta for timestamps, XOR for float values). We achieve an average compression factor of **10× to 35×** compared to text-based JSON.

Scenario Parameters

• Active Devices ($N$): 1,000,000
• Transmission Interval ($T$): 10 seconds (1 reading per device every 10s)
• Average JSON Text Payload size: 180 bytes
• Equivalent Binary Protobuf size: 32 bytes
• Average TSDB Compressed record size on disk: 1.5 bytes

Worked Calculations: Network & Storage Footprint

How to debug & inspect it

To debug IoT ingestion pipelines, monitor the message broker lag (to ensure consumers are keeping pace with device writes) and inspect binary payloads directly.

Use the guide below to address common failures in high-throughput append-only pipelines:

CREATE TABLE metric_hourly_rollups (
    device_id VARCHAR(64) NOT NULL,
    metric_id INT NOT NULL,
    hour_timestamp TIMESTAMP NOT NULL,
    reading_count INT NOT NULL,     -- Tracks number of raw events rolled up
    min_value DOUBLE PRECISION,     -- Minimum value in the hour
    max_value DOUBLE PRECISION,     -- Maximum value in the hour
    avg_value DOUBLE PRECISION,     -- Arithmetic mean
    PRIMARY KEY (device_id, metric_id, hour_timestamp)
);

Sources & further reading

• Gorilla Paper (Facebook) — A Fast, In-Memory Time Series Database — describes double-delta timestamp compression and XOR floating point compression
• TimescaleDB Architecture — Hypertables & Chunking — explanation of memory-resident index scaling for time-series
• MQTT Specification & Protocols — explaining the light-overhead design parameters for constrained networks