Platform & API Product Engineering · Lesson 08

Usage metering & monetization

Every missed billable event is lost revenue; every double-counted event produces an angry customer and a chargeback. Building metering that is simultaneously durable, accurate, and queryable — at millions of events per day — requires the same engineering rigor as your most critical data pipeline.

Why metering is harder than logging

Application logging tolerates loss: a missing log line means a gap in your dashboard, not a business error. Metering is different. A missed usage event means a customer goes unbilled for real work your system performed — lost revenue that compounds every billing cycle. A double-counted event means an overcharge, a support ticket, a refund, and erosion of trust. The asymmetry of consequences forces metering to be designed like a financial system, not a logging system.

The second challenge is scale. Stripe processes millions of payment events daily. Twilio sends billions of SMS per year. AWS meters hundreds of services per second across millions of customers. At those volumes, even a 0.01% error rate is operationally significant. The solutions that work in production — durable queues, pre-aggregated counters, idempotency keys — are not accidental. They emerged from painful lessons about what breaks at scale.

Think of the metering pipeline as a bank's double-entry ledger: every transaction is recorded first in a durable journal (the queue), then summarized in account balances (the counters). You can reconstruct the balances from the journal. You cannot reconstruct the journal from the balances. That asymmetry shapes every architectural decision below.

The metering pipeline: five stages

Stage 1 — emit a usage event per billable call

Every billable action — an API call, a sent message, a processed transaction — produces one usage event. The event schema must include an event_id (a UUID or content-hash used as the idempotency key), account_id, metric_name, quantity, and timestamp. Metadata fields (region, product_tier, API version) enable later analytics but are not required for billing correctness.

# Usage event schema — emitted per billable action
{
  "event_id":        "evt_a1b2c3d4e5f6",    // idempotency key — UUID v4 or content hash
  "account_id":      "acct_42",
  "metric_name":     "sms_sent",
  "quantity":        1,                       // or segment_count for multi-part SMS
  "timestamp":       "2026-06-20T10:30:00.123Z", // ISO 8601 with ms precision
  "metadata":        { "region": "us-east-1", "tier": "growth" }
}

The emission must be fire-and-forget from the request path. If the meter write is synchronous (the API call blocks waiting for the billing system to acknowledge), a billing system outage becomes an API outage. Emit to the queue asynchronously; the request path should never wait on the metering infrastructure.

Stage 2 — durable queue (the journal)

The queue — typically Kafka, but AWS Kinesis or Google Pub/Sub serve the same purpose — is the metering system's source of truth. It buffers events between emission and processing, provides durability (events survive consumer crashes), and enables replay (re-process events from any offset to correct bugs or backfill missing data).

The critical property is at-least-once delivery: Kafka guarantees every acknowledged event is stored until consumed, but a consumer crash after processing but before committing its offset will replay the event. This is by design — it means you never lose revenue — but it requires the aggregator to handle duplicates. See Stage 3.

Kafka's retention period (typically 7 days, extendable to 30+) serves as the raw event store for audit and late-event replay. This is different from the counter store: the raw events answer "show me every SMS sent on June 20" while the counters answer "how many SMS did account 42 send in June?"

Stage 3 — aggregate: dedup + count

The aggregator consumer reads events from Kafka, deduplicates them using the event_id, and increments per-account per-metric per-period counters. In Redis, this looks like:

# Aggregator pseudocode — for each event from Kafka
def process_event(event):
    seen_key    = "seen:" + event["event_id"]
    counter_key = "usage:" + event["account_id"] + ":" + period(event["timestamp"]) \
                  + ":" + event["metric_name"]

    # Atomic dedup check + increment using Lua script
    lua_script = """
      local seen_key   = KEYS[1]
      local ctr_key    = KEYS[2]
      local qty        = tonumber(ARGV[1])
      local ttl_seen   = tonumber(ARGV[2])  -- e.g. 2592000 (30 days)

      if redis.call('EXISTS', seen_key) == 1 then
        return 0  -- already processed; skip
      end
      redis.call('SETEX', seen_key, ttl_seen, '1')  -- mark as seen
      redis.call('HINCRBY', ctr_key, 'total', qty)  -- increment counter
      return 1  -- processed
    """
    result = redis.eval(lua_script, [seen_key, counter_key],
                        [event["quantity"], 2592000])

    if result == 0:
        LOG("duplicate event skipped: " + event["event_id"])
    else:
        kafka.commit_offset()  # only commit after successful processing

Stage 4 — usage API (customer-facing)

Customers need to see their own consumption — not just at invoice time, but on-demand. A well-designed usage API serves this from the counter store (Redis), giving near-real-time data without expensive raw-event scans:

# Usage API — GET /v1/usage
GET /v1/usage?metric=sms_sent&period=2026-06&account_id=acct_42
Authorization: Bearer {customer_api_key}

# Response
{
  "account_id":    "acct_42",
  "metric":        "sms_sent",
  "period":        "2026-06",
  "consumed":      8432,
  "quota":         10000,
  "quota_remaining": 1568,
  "as_of":         "2026-06-20T10:30:45Z"   // when the counter was last updated
}

The implementation: a single HGET usage:acct_42:2026-06:sms_sent total Redis call. Compare that to scanning raw events from Kafka — which would require reading gigabytes of data for a long-running account. Pre-aggregated counters are why the usage API is fast.

Stage 5 — billing export

At the end of each billing period (typically midnight UTC), a billing exporter reads the final counter values for every active account and submits invoice line items to the billing system. For Stripe Billing metered subscriptions, this means calling the meter events API or the usage records endpoint. For internal systems, it means writing records to a billing database table.

The exporter should reconcile: compare the counter total to a raw-event count (by scanning Kafka or a separate raw-event database) for a sample of accounts. Persistent discrepancies indicate a dedup bug or an aggregator lag that needs investigation before the invoice is sent.

Accuracy under retries and failures

Late events and reconciliation

Network partitions and Kafka consumer lag mean some events arrive after the billing period closes. A message sent at 23:59:58 on June 30 might not reach the aggregator until July 1 00:00:03. Three strategies handle this:

1. Use event timestamp, not arrival timestamp. The counter key includes the period derived from event.timestamp, not now(). An event arriving late is credited to the correct billing period.
2. Keep counters writable for a grace window. Hold the billing period open for 5–15 minutes after midnight before exporting. Most late events arrive within seconds; a 5-minute grace window catches nearly all of them.
3. Reconciliation run. For high-value accounts, scan the raw events from Kafka for the period and compare the sum to the counter value. If they diverge beyond a threshold, investigate before sending the invoice. This is the equivalent of a bank's end-of-day balancing run.

Quotas, plans, and overage

Soft vs hard limits

A hard limit blocks requests with a 429 once usage reaches the quota ceiling. A soft limit allows requests to continue — emitting a warning to the customer and flagging the account for overage billing — but does not block. The right choice depends on the product contract: infrastructure APIs (Twilio SMS, AWS Lambda) typically use hard limits to protect the platform; SaaS APIs often prefer soft limits with graduated billing because an unexpected 429 in a production workflow is worse for customer trust than an unexpected charge that can be explained.

Both types connect to the rate limiter (see plat-01): the quota counter and the rate limiter both live in Redis, and the gateway reads both on every request. Rate limiting governs how fast a customer sends requests; quota governs how many over a billing period. They are different dimensions of the same enforcement problem.

Pricing models: metered, seat, tiered

Metered pricing bills per unit of consumption (per SMS, per API call, per GB processed). Every unit must be metered. This lesson covers metered pricing because it requires the most infrastructure — every call is billable and must be counted accurately.

Seat pricing bills per active user regardless of usage. Metering is simpler (count provisioned users, not calls), but you still need usage data to identify inactive seats and justify renewals.

Tiered pricing combines a base quota (included in the plan) with overage billing above it. The quota counter serves both: it drives hard/soft limit enforcement during the month and provides the final total for overage calculation at period close.

Under the hood: from API call to billing record

Trace a single SMS-send request through the entire metering pipeline, including a retry scenario:

By the numbers: 100 million billable events per day

Scale the pipeline to understand where storage and compute cost actually lives.

Raw event storage in Kafka (modeled):

Pre-aggregated counter storage in Redis (modeled):

Dedup seen-set size (modeled):

Governing formula — when pre-aggregation pays off:

Aggregation window decision: Real-time stream aggregation (Flink, a Redis consumer) keeps counters current within seconds — the right choice for customer-facing usage dashboards. Hourly batch jobs reduce infrastructure complexity but mean the usage API is 0–60 minutes stale, acceptable for billing exports but frustrating for customers debugging unexpected quota exhaustion. Daily batch jobs are only appropriate if you never expose real-time usage to customers and billing periods are monthly.

Pros and cons: the four key trade-offs

How real platforms do it

Stripe Billing metered subscriptions use a meter events API: you report usage by pushing events to Stripe's API (one call per billable action, with an idempotency key to prevent double-counting). Stripe aggregates these events and includes them in the customer's upcoming invoice. Stripe also supports a "billing meter" resource that lets you configure aggregation mode (sum, max, last value) per metric. See Stripe usage-based billing docs.

AWS Marketplace Metering Service requires SaaS products listed on the Marketplace to report metered usage by calling the MeterUsage API. AWS enforces that each call is idempotent (via a UsageDimension + timestamp key) and provides a 1-hour grace window for late reporting. Usage is aggregated per customer, per dimension, per hour. See AWS Marketplace metering integration docs.

Twilio meters at the point of delivery: an SMS or call is billable when the carrier confirms delivery (or attempt). Each message has a unique SID that serves as the idempotency key. Twilio's usage records API exposes per-account, per-resource, per-period consumption — the customer-facing equivalent of the usage API described in Stage 4. See Twilio pricing and usage model.

Sources & further reading

• Stripe — Usage-based billing and metered subscriptions
• AWS Marketplace — SaaS metering integration
• Twilio — Pricing and usage model
• Apache Kafka — Design: consumer pull and durability
• Lesson rel-02 — Idempotency
• Lesson rel-10 — Event-driven architecture & Pub/Sub
• Lesson prep-05 — Business sense & pricing