Platform & API Product Engineering · Lesson 09

Capstone: design a developer platform (Stripe/HubSpot-style)

Every concept in this module — app models, key rotation, scoped auth, nested rate limits, webhook delivery, error envelopes, tenant isolation, usage metering — was a separate instrument. This lesson assembles them into one instrument that plays. You will design a production-grade developer platform from requirements through trade-offs to a concrete API surface and the numbers that show where it breaks.

1 — Requirements

Good design begins with precise requirements. Vague requirements produce systems that satisfy no one; over-specified requirements close off the right trade-offs. Start by separating what the platform must do from what constraints bound how it does it.

Functional requirements

• App management: each account can create up to 20 private apps. A private app is an isolated principal within the account — it has its own identity, its own keys, and its own permission surface. Apps cannot see each other's keys or webhook history.
• API key lifecycle: every app has at least one active API key at all times. Keys can be created, rotated (overlap period during which old and new key both work), and revoked. A revoked key must be rejected within a bounded propagation window, not eventually.
• Webhook subscriptions: each app subscribes to a named set of event topics (e.g. contact.created, deal.stage_changed). The platform delivers signed payloads to one or more HTTPS endpoints registered per app. Subscriptions are scoped — an app that does not subscribe to billing.invoice.paid never receives that event, even if the account generates it.
• Scopes / permissions: each app is granted a set of OAuth-style scopes at install time. A key from an app that holds crm.contacts:read but not crm.contacts:write is rejected on mutation attempts — not just for billing, but at the authorization layer.
• Per-app rate limit: the platform enforces a request rate limit at app granularity. A runaway app cannot exhaust the account's capacity.
• Per-account daily cap: in addition to per-app rate limits, each account has a hard daily API call cap that aggregates across all its apps. Once the cap is reached, all apps in that account receive 429 until the UTC-midnight reset.
• Reliable webhook delivery: events are delivered at least once. Non-200 responses and timeouts trigger retries with exponential backoff. Deliveries are signed so recipients can verify authenticity without trusting the source IP.
• Consistent error model: every error — auth failure, rate limit, validation, internal fault — uses the same JSON envelope. Clients parse one shape, not a different one per error class.
• Usage metering: every billable API call is recorded for downstream billing. The metering record is written as part of the request path — not asynchronously after the fact — so billing is exact, not approximate.

Non-functional requirements

2 — Design decisions

Each decision below is a module concept applied to the specific requirements above. For each, there is a key trade-off that you must be able to state — not just describe the mechanism.

Decision 1 — App and key model

The plat-06 lesson established that a developer platform key is not just an auth credential — it is the anchor for every other per-app concern: rate limit counters, webhook subscriptions, scope grants, and metering buckets all hang off the app's identity. The key embeds or resolves to three identifiers: account_id, app_id, and key_id. The gateway validates the key, extracts these three, and injects them as context for every downstream decision.

The 20-app cap is a product decision, not a technical limit. The platform can technically support any number of apps per account — the rate-limit key space and the metering aggregation are both keyed by app_id and scale horizontally. The cap exists so that accounts do not create "shadow apps" to route around per-app rate limits, and so that the daily-cap math remains meaningful (a per-account cap only makes sense if the number of apps is bounded). Enforcing the cap is a write-path check: CREATE APP counts existing apps and rejects at 20 with a 400 error.

Key rotation trade-off: the clean solution is to let old and new keys overlap for a configurable window (default: 1 hour). During overlap, both keys are valid; the gateway checks a small local cache of active key IDs. The trade-off is cache coherence: when a key is revoked, the platform must propagate the revocation within a bounded window (the SLA is 60 seconds, not eventual). This requires either a short cache TTL (60 s) or a pub/sub invalidation channel from the key store to all gateway replicas. Short TTL is simpler to operate; pub/sub invalidation is faster. Most platforms at this scale use both: a 60 s TTL as a safety net and a pub/sub channel for instant revocations.

Decision 2 — Scopes and install-time authorization

The plat-07 lesson covered OAuth-style scopes for platform apps. The scope model here has one platform-specific wrinkle: scopes are granted at app install time and are immutable per key. If an integration needs an additional scope, it must re-install and get a new key with the expanded grant — it cannot add scopes to an existing key. This is a deliberate design choice from the account-holder's perspective: a key you granted last year should not be able to request new permissions without your explicit re-consent.

Trade-off: static vs. dynamic scope grants. Static grants (baked at install time) are auditable, predictable, and immune to confused-deputy attacks where a compromised key tries to escalate its own permissions. Dynamic grants (scope expansion on-demand) reduce friction for iterative integrations. Stripe uses static grants for restricted keys; HubSpot requires a new OAuth authorization flow for scope expansion. For a platform where security is the product (accounts trusting third-party integrations with their customer data), static grants are the right default.

Scope enforcement happens in the gateway, before the request reaches any service. The gateway reads the scope list from the resolved key context and compares it to a route-to-scope map: POST /v1/contacts → [crm.contacts:write]. A key that lacks the required scope receives a 403 with "code": "insufficient_scope" before any service is invoked.

Decision 3 — Nested rate limits and the daily cap

The plat-01 lesson laid out the nested-bucket model: every request must pass two independent rate-limit checks before it is allowed. Here the hierarchy has three levels:

1. Per-app burst limit — token bucket, enforced in real time. Default: 100 req/s. A runaway script in one app cannot flood the gateway. Checked first because it is the tightest and most frequently triggered.
2. Per-account burst limit — token bucket, enforced in real time. Default: 500 req/s aggregate across all apps. Prevents an account from spinning up 20 apps each doing 100 req/s simultaneously to achieve an effective 2,000 req/s.
3. Per-account daily cap — sliding or fixed window counter, daily granularity. Default: 1,000,000 calls/day. This is a billing-tier enforcement. When the cap is exhausted, all requests from all apps in the account receive 429 until UTC midnight. The counter is stored in Redis with a TTL aligned to UTC midnight.

The gateway checks level 1 first (cheapest — only one Redis key per app), then level 2 (one Redis key per account), then level 3 (daily counter). A request that passes all three is allowed; the response carries all three sets of rate-limit headers so a client SDK can observe which limit is closest to exhaustion.

Trade-off: where to enforce the daily cap. The daily cap counter is the most expensive to maintain at 8,000 req/s peak: every request increments it and reads its current value. Using a single Redis key per account for the daily cap creates a hot key problem if one account runs 8,000 req/s — all of those increments funnel to one slot. The solution is a local-increment / periodic-flush pattern: each gateway replica maintains a local counter shard per account and flushes to Redis every 100 ms. The in-Redis counter is approximately accurate for enforcement; the exact billing count comes from the metering store. At 100 ms flush intervals, the maximum error on the daily cap is at most (8,000 × 0.1) = 800 calls above the cap before a gateway replica catches up — less than 0.1% of the 1M daily cap. This is an acceptable enforcement window for a soft financial constraint.

Decision 4 — Webhook delivery subsystem

The plat-02 lesson described the full delivery pipeline. For this platform, the webhook system has two properties that interact: fan-out and scoped subscriptions. When an event is generated, the platform must determine which apps across which accounts have subscribed to that event type — then enqueue one delivery task per (app, endpoint) tuple. At 20 M deliveries/day, this fan-out step is the most CPU-intensive part of the webhook system.

Subscription index: the platform maintains a materialized index: event_type → [(account_id, app_id, endpoint_url, signing_secret)]. When an event fires, a single lookup against this index returns all delivery targets. The index is maintained in a fast read store (Redis or an in-process cache backed by a relational store). Invalidation happens on subscription create/update/delete via the same pub/sub channel used for key invalidation.

Delivery reliability: each delivery task is written to a durable queue (Kafka or a Postgres-backed job queue) before the fan-out process returns. The queue is the durability guarantee — even if the delivery worker crashes, the task survives. Workers consume from the queue, attempt the HTTPS delivery, and either acknowledge (success) or nack with a delay (retry). Retries use exponential backoff capped at 24 hours, after which the delivery is marked permanently failed and a webhook.delivery.failed event is written to the account's event log.

Trade-off: Kafka vs. Postgres job queue. Kafka gives higher throughput and native consumer-group parallelism but requires separate infrastructure and adds operational complexity. A Postgres-backed job queue (using SELECT FOR UPDATE SKIP LOCKED) is operationally simpler and sufficient up to ~5,000 jobs/s — which covers the 230 deliveries/s average load with significant headroom for spikes. At 20 M deliveries/day the Postgres queue is the right choice; at 200 M deliveries/day (10× growth) Kafka becomes necessary. See plat-02 for the delivery queue mechanics.

Decision 5 — Error model

The plat-03 lesson established the standard error envelope. Every error from this platform — regardless of which service generated it — is normalized to the same JSON shape at the gateway before it reaches the client:

{
  "error": {
    "code":    "rate_limit_exceeded",          // machine-readable; stable across versions
    "message": "App rate limit reached: 100 req/s. Retry after 12 s.",
    "status":  429,
    "type":    "rate_limit_error",               // error class; maps to documentation section
    "param":   null,                             // populated for validation errors: which field
    "request_id": "req_01J9W3KZR4TY8X2N6M5L7P"   // traceable in logs; always present
  }
}

Trade-off: normalized gateway vs. pass-through service errors. Normalizing at the gateway means every service can return its own internal error format — only the gateway translation layer needs to know the canonical envelope. The downside is that the gateway must map unfamiliar error shapes (e.g. a database timeout that surfaces as a Go context deadline exceeded) to the correct canonical code. This mapping is a small translation table, not a free-form transformation — each internal error code maps to exactly one canonical code and HTTP status. Unmapped errors become 500 / internal_error with the request_id for tracing.

Decision 6 — Tenant isolation

The plat-05 lesson described the isolation models for multi-tenant platforms. For this platform, tenant isolation operates at three layers simultaneously:

• Data isolation: every row in the platform's data store carries an account_id column, and every query is filtered by it. No cross-account read is possible through the application layer. The only way to access another account's data is through the admin service, which has separate credentials and a mandatory audit log entry on every use.
• Rate-limit isolation: the per-app and per-account buckets ensure one tenant's traffic spike does not consume shared capacity. The daily cap is the final backstop.
• Webhook isolation: event fan-out uses the subscription index keyed by (event_type, account_id) — events generated by account A are never delivered to apps registered in account B, even if both subscribe to the same event type.

Trade-off: shared infrastructure vs. dedicated resources. All 50,000 accounts share the same Redis cluster, gateway fleet, and webhook worker pool. This is economically necessary and is standard practice (Stripe, HubSpot, and GitHub all run shared-infrastructure multi-tenancy at this scale). The risk is noisy-neighbor effects. Mitigation: per-app and per-account rate limits cap any single tenant's resource consumption; the Redis cluster uses consistent hashing so no single shard holds all keys for a popular account; webhook workers use per-account FIFO lanes so a backlogged account does not starve other accounts' delivery queues.

Decision 7 — Usage metering

The plat-08 lesson covered the metering pipeline in detail. The critical constraint for this platform is billing durability: if a call is made, it must be billed. This means the metering write must complete before the response is returned to the client — not asynchronously afterward. The mechanism is the hybrid counter described in Decision 3: an atomic Redis increment is the synchronous write (fast, durable within the Redis cluster's replication SLA); background flush jobs serialize snapshots to a billing-grade relational store every 60 seconds. The billing store is the invoice source; the Redis counter is the enforcement source. They diverge by at most one flush interval (60 s × throughput), which is acceptable for billing reconciliation but must be documented in the platform's billing terms.

Architecture

The platform's request path passes through seven discrete checkpoints before reaching any product service. Understanding each checkpoint is the difference between guessing where a failure comes from and knowing.

Request lifecycle through the gateway

For any individual request, the gateway executes a strict linear sequence. Understanding the sequence is how you debug: a 401 means auth failed; a 403 means auth passed but scope check failed; a 429 means auth and scope passed but a rate limit or daily cap rejected the request. No 429 is ever returned before the key is valid.

3 — The API model

The platform's public surface consists of four resource families: apps, keys, webhook endpoints, and event subscriptions. All other resources (contacts, deals, billing records) belong to the product services — the platform surface is the configuration layer that wraps them.

App CRUD

# Create a private app (max 20 per account)
POST /v1/apps
Authorization: Bearer <account-level-key>
Content-Type: application/json

{
  "name":   "Nightly CRM Sync",
  "scopes": ["crm.contacts:read", "crm.deals:read"],
  "description": "Reads contacts and deals nightly for data warehouse sync"
}

─── 201 Created ───────────────────────────────────────────────────
{
  "id":          "app_01J9W3KZR4TY8X2N6M5L7P",
  "account_id":  "acct_9pXwL4mQ",
  "name":        "Nightly CRM Sync",
  "scopes":      ["crm.contacts:read", "crm.deals:read"],
  "status":      "active",
  "created_at":  "2025-11-15T09:00:00Z",
  "rate_limits": {
    "per_app_rps":   100,
    "per_acct_rps":  500,
    "daily_cap":    1000000
  }
}

# List apps (paginated)
GET /v1/apps?limit=20&cursor=app_01J9W3…

# 400 when the 20-app cap is hit
{
  "error": {
    "code":    "app_limit_exceeded",
    "message": "This account has reached the maximum of 20 private apps.",
    "status":  400,
    "type":    "validation_error",
    "param":   null
  }
}

Key management — create, rotate, revoke

# Create a new key for an app
POST /v1/apps/app_01J9W3KZR4TY8X2N6M5L7P/keys

─── 201 Created ───────────────────────────────────────────────────
{
  "id":           "key_3mZqR7tXv",
  "secret":       "hbsp_live_v2_a1B2c3D4e5F6g7H8i9J0k_acct9pXwL4_app01J9W3_key3mZqR7",
  "created_at":   "2025-11-15T09:01:00Z",
  "last_used_at": null
}
// secret shown ONCE at creation time; store it immediately

# Rotate: creates new key, old key remains valid for overlap_seconds (default 3600)
POST /v1/apps/app_01J9W3…/keys/key_3mZqR7tXv/rotate
{ "overlap_seconds": 3600 }

─── 201 Created ───────────────────────────────────────────────────
{
  "new_key": { "id": "key_9nYsQ4uWm", "secret": "hbsp_live_v2_…" },
  "old_key": {
    "id":         "key_3mZqR7tXv",
    "expires_at": "2025-11-15T10:01:00Z"   // overlap window end
  }
}

# Revoke immediately (propagated within 60 s to all gateway replicas)
DELETE /v1/apps/app_01J9W3…/keys/key_3mZqR7tXv

─── 200 OK ────────────────────────────────────────────────────────
{ "id": "key_3mZqR7tXv", "status": "revoked", "revoked_at": "2025-11-15T09:45:00Z" }

Webhook endpoint configuration

# Register a webhook endpoint and subscribe to event topics
POST /v1/apps/app_01J9W3…/webhooks
{
  "url":         "https://integrations.acme.com/hs-hooks",
  "event_types": ["contact.created", "contact.updated", "deal.stage_changed"],
  "description": "Sync CRM events to Acme data warehouse"
}

─── 201 Created ───────────────────────────────────────────────────
{
  "id":             "wh_7kLpM2nQr",
  "url":            "https://integrations.acme.com/hs-hooks",
  "event_types":    ["contact.created", "contact.updated", "deal.stage_changed"],
  "signing_secret": "whsec_K9mP3rT7vX2qL5nY1wZ8…",  // shown once; use for HMAC verification
  "status":         "active",
  "created_at":     "2025-11-15T09:05:00Z"
}

# Example delivery (signed)
POST https://integrations.acme.com/hs-hooks
X-HubSpot-Signature-v3: t=1731660305,v3=sha256=3b4f…
X-HubSpot-Request-Id: req_8xNtR5kZv
Content-Type: application/json

{
  "event_id":    "evt_2YhJp6mWq",
  "event_type":  "contact.created",
  "occurred_at": "2025-11-15T09:10:00Z",
  "account_id":  "acct_9pXwL4mQ",
  "app_id":      "app_01J9W3KZR4TY8X2N6M5L7P",
  "object": {
    "type": "contact",
    "id":   "ct_5pQnX9rYz",
    "properties": { "email": "alice@acme.com", "firstname": "Alice" }
  }
}

Rate-limit response headers on every request

─── 429 rate limit — per-app burst ────────────────────────────────
HTTP/1.1 429 Too Many Requests
X-RateLimit-App-Limit: 100
X-RateLimit-App-Remaining: 0
X-RateLimit-App-Reset: 1731660362
Retry-After: 2

{
  "error": {
    "code":       "rate_limit_exceeded",
    "message":    "App rate limit: 100 req/s. Reset in 2 s.",
    "status":     429,
    "type":       "rate_limit_error",
    "limit_type": "per_app",          // which of the three limits triggered
    "param":      null,
    "request_id": "req_01J9W3KZR4TY8X2N6M5L7P"
  }
}

─── 429 daily cap exhausted ───────────────────────────────────────
{
  "error": {
    "code":       "daily_cap_exceeded",
    "message":    "Daily API cap of 1,000,000 calls reached. Resets at 2025-11-15T00:00:00Z (UTC midnight).",
    "status":     429,
    "type":       "rate_limit_error",
    "limit_type": "daily_cap",
    "param":      null,
    "request_id": "req_7pTmN8qXv"
  }
}

4 — Evaluation & by the numbers

Design decisions are only real when they survive contact with concrete numbers. This section traces the load through the system, identifies where the first bottleneck appears, and works out the math on the two most important limits: the 20-app cap and the daily cap.

The 20-app, 1M-call-per-day math: which limit binds first?

An account has 20 apps, each with the default per-app rate limit of 100 req/s. If all 20 run simultaneously at their per-app limit:

Per-app limit:         100 req/s per app
Apps at full throttle: 20 apps
Combined throughput:   20 × 100 = 2,000 req/s

But the per-account burst limit: 500 req/s
→ Per-account limit BINDS before per-app can be fully exploited across all 20 apps

At 500 req/s sustained, daily call volume:
  500 req/s × 3,600 s/hr × 24 hr = 43,200,000 calls/day
  43.2M >> 1M daily cap
→ The DAILY CAP binds long before the burst rate limit is a concern

Time to exhaust the 1M daily cap at sustained 500 req/s:
  t = 1,000,000 / 500 = 2,000 s = 33.3 minutes

Time to exhaust at the more typical 100 req/s (one busy app):
  t = 1,000,000 / 100 = 10,000 s = 2.78 hours
  → A single app running at its burst limit all day consumes the full daily cap in ~3 hours.
  → All other apps in the account are locked out for the remaining ~21 hours.

Implication: the daily cap is a HARD account-level governance tool, not just billing.
Apps should implement exponential backoff when Remaining → 0 at the daily layer.

Webhook fan-out volume and queue sizing

At 20 M webhook deliveries/day, understand the distribution between generation and delivery:

The bottleneck is not queue throughput — Postgres can handle this insert rate. The bottleneck is outbound connection concurrency: 740 simultaneous open HTTPS connections from the worker pool to customer endpoints. At this scale, workers need a connection pool with per-domain connection limits (to avoid overwhelming any single customer endpoint), and a circuit breaker per endpoint to avoid wasting worker slots on consistently failing destinations. See plat-02 for the circuit breaker pattern on webhook workers.

Metering volume and flush math

API calls/day:                  100,000,000
Peak calls/s:                   ~8,000 req/s (7× daily avg)
Metering writes/s (Redis INCR): 8,000  ← same as call volume (one per request)
Redis INCR throughput per node: ~100,000/s (single-threaded pipeline)
  → 1 Redis node handles peak metering load with 92% headroom
  → Shard by account_id across 4 Redis nodes for isolation; no single node hot

Flush interval to billing DB:   60 s
Max calls un-flushed at peak:   8,000 × 60 = 480,000 calls per flush batch
Billing DB rows written/day:    50,000 accounts × (24 × 60 / 1) = 1 row/flush/account
  = 50,000 × 1,440 flushes/day = 72,000,000 billing DB inserts/day
  → Use upsert: UPDATE counter WHERE date=today AND account_id=X, not 72M inserts
  → Actual write: UPDATE metering_daily SET calls = calls + <batch> WHERE ...
  → One row per (account_id, app_id, date) updated every 60 s: manageable

Platform-scale system limits trace

How real platforms do it

The design above is not hypothetical — it is a synthesis of documented practices from three platforms that have operated at comparable or larger scale for years.

Sources & further reading

• HubSpot — Private Apps: creating, scoping, and managing private apps
• HubSpot — API Usage & Limits: daily limits, burst limits, and monitoring headers
• HubSpot — Validating Webhook Requests: v3 signature scheme including timestamp
• Stripe — Restricted API Keys: per-resource scope grants and enforcement
• Stripe — Rate Limits: token-bucket model, 429 behavior, and retry guidance
• Stripe — Webhook Signatures: HMAC-SHA256, timestamp, replay prevention
• Stripe — Idempotent Requests (referenced from cs-12-stripe.html for context)
• GitHub REST API — Rate Limits: primary and secondary limits, x-ratelimit-* headers
• GitHub Apps — Installation tokens, scope grants, and re-authorization for scope expansion
• GitHub — Validating Webhook Deliveries: X-Hub-Signature-256 implementation
• Redis — Lua Scripting: atomic multi-key operations via EVAL
• PostgreSQL — SELECT FOR UPDATE SKIP LOCKED: the foundation of Postgres-backed job queues