Platform & API Product Engineering · Lesson 01

Nested & Tiered Rate Limits

A single inbound request does not face one rate limit — it faces a stack of them, evaluated simultaneously. A per-key burst limit, a per-app sustained rate, and a per-org daily quota all fire in parallel, and a request is rejected the moment any one of them trips. Getting this composition right is harder than any individual algorithm.

Why a single rate limit is never enough

A developer integrating with your API might issue a burst of 200 requests in a second testing their new feature — that is a per-key burst concern. Their app, overall, sustains 80 requests per second across all keys — that is a per-app throughput concern. Their whole organization, across 20 apps, consumes 900,000 API calls today — that is a billing and capacity concern. These are three distinct failure modes at three distinct time scales. A single token bucket conflates them.

Think of it like airport security. Your boarding pass (API key) is checked at the gate — that is the burst check, closest to the individual. The airline's total load on a given aircraft (per-app rate) limits how many passengers they can board per minute regardless of individual keys. And the airport has a daily throughput limit before ground operations degrade (the org daily cap). Passing one check does not mean you pass the others.

Real platforms formalize this with a quota hierarchy: an organization (org) owns one or more apps; each app has one or more API keys. Limits cascade downward — an org's daily cap is shared across all its apps, each app's sustained rate is shared across all its keys, and each key has its own burst envelope.

The layered gate model

Visualize a request passing through three gates in sequence. All three gates evaluate simultaneously (in one atomic Redis operation, as you will see), but conceptually each answers a different question.

The data model: three types of Redis state

Each layer uses a different data structure because each operates at a different time scale and with a different eviction strategy.

Layer 1 — Per-key token bucket (burst, per-second)

A Redis hash per API key holding two fields: the current token count and the timestamp of the last refill. This is the same lazy-refill token bucket from Lesson rel-03. The key uses the API key string directly so each key is isolated.

Layer 2 — Per-app token bucket (sustained rate, also per-second)

An identical hash structure, but keyed on the app ID rather than the individual API key. All keys belonging to the same app share this single bucket. If app appX has 10 API keys and each key fires at 20 req/s, they collectively drain the app's 100 req/s sustained bucket together.

Layer 3 — Per-org daily quota counter (calendar day)

A plain Redis integer counter keyed on org ID and the calendar date. This is a fixed-window counter with a TTL set to expire at midnight of the next day — not a fixed TTL from creation, but a deadline TTL calculated from the current time. The distinction matters: a key created at 23:00 must expire in 1 hour, not 24 hours.

Timezone: UTC vs account timezone

The date key in quota:acct:{org}:{YYYYMMDD} should almost always use UTC for the date string, even if your platform shows "resets daily" to customers in their local timezone. Using account timezone for the Redis key creates a fragmented keyspace (the same org has a different daily key in different time zones) and makes the TTL calculation per-org rather than global. The practical approach: store quota in UTC, but display reset times to customers in their configured timezone.

Under the hood: the atomic multi-limit check

The composition problem has a non-obvious failure mode: if you check each limit in a separate Redis command, you can partially consume one limit before discovering another is exhausted. Imagine checking the key burst bucket first (it passes and you deduct a token), then checking the app bucket (it is full — you must reject). You have now consumed a token from the key bucket for a request that was never served. Repeated partial-consumes slowly bleed quota from the higher layers without the client receiving any successful responses.

The fix is to run all checks in a single Lua script. Redis executes the entire Lua script atomically — no other command runs between steps. The script reads all three counter states, checks all three limits, and only decrements any of them if all three pass.

The Lua script: check-all-then-commit

-- Atomic multi-limit check: per-key burst + per-app sustained + per-org daily
-- KEYS[1] = rl:key:{api_key}
-- KEYS[2] = rl:app:{app_id}
-- KEYS[3] = quota:acct:{org_id}:{YYYYMMDD}
-- ARGV[1] = key_capacity (tokens), ARGV[2] = key_refill_rate (tokens/ms)
-- ARGV[3] = app_capacity,          ARGV[4] = app_refill_rate (tokens/ms)
-- ARGV[5] = org_daily_cap,         ARGV[6] = now_ms (from KEYS-side caller)
-- ARGV[7] = secs_until_midnight (for TTL on org key)

local key_bkt  = KEYS[1]
local app_bkt  = KEYS[2]
local org_ctr  = KEYS[3]

local now      = tonumber(ARGV[6])

-- ① READ all three states (no writes yet)
local kt  = redis.call('HMGET', key_bkt, 'tokens', 'ts')
local at  = redis.call('HMGET', app_bkt, 'tokens', 'ts')
local org = tonumber(redis.call('GET', org_ctr)) or 0

-- ② Compute token-bucket state for key layer
local k_tokens    = math.min(tonumber(ARGV[1]),
    (tonumber(kt[1]) or tonumber(ARGV[1])) +
    (now - (tonumber(kt[2]) or now)) * tonumber(ARGV[2]))

-- ③ Compute token-bucket state for app layer
local a_tokens    = math.min(tonumber(ARGV[3]),
    (tonumber(at[1]) or tonumber(ARGV[3])) +
    (now - (tonumber(at[2]) or now)) * tonumber(ARGV[4]))

-- ④ CHECK all three limits — NO writes yet
if k_tokens < 1 then
  return {0, 'key', k_tokens}   -- rejected at key layer
end
if a_tokens < 1 then
  return {0, 'app', a_tokens}   -- rejected at app layer
end
if org >= tonumber(ARGV[5]) then
  return {0, 'org', org}        -- rejected at org daily cap
end

-- ⑤ ALL checks passed — WRITE (commit all three decrements atomically)
redis.call('HSET', key_bkt, 'tokens', k_tokens - 1, 'ts', now)
redis.call('HSET', app_bkt, 'tokens', a_tokens - 1, 'ts', now)
local new_org = redis.call('INCR', org_ctr)
if new_org == 1 then                         -- first call today: set TTL
  redis.call('EXPIRE', org_ctr, tonumber(ARGV[7]) + 300)
end

return {1, 'ok', k_tokens - 1, a_tokens - 1, tonumber(ARGV[5]) - new_org}

Calendar-day window: the TTL arithmetic

The org daily counter expires at UTC midnight, not N hours after creation. Using a fixed-TTL EXPIRE would let a key created at 18:00 run until 18:00 the next day — not the same billing day. The correct approach is to compute the remaining seconds until the next UTC midnight and use that as the TTL.

Quota inheritance: org → app → key

The three limits are not independent figures pulled from three separate config files. They form an inheritance tree. An org has a total daily cap; each app is provisioned a fraction of the org's sustained rate; each key under an app gets a personal burst envelope. The numbers should be consistent: if an org has 20 apps each running at full sustained rate, the aggregate must not exceed the org cap.

GraphQL query cost weights: a fourth dimension

REST endpoints are roughly uniform in cost: one request = one token consumed. GraphQL mutations and queries vary enormously — a deeply nested query that joins four resolver trees is worth far more than a single field fetch. Shopify and GitHub both expose this as a query cost model: each field has a cost weight, the resolver sums the weights before execution, and the rate limiter deducts that many tokens from the bucket rather than one. This is a fourth layer that applies only to the GraphQL surface.

# GraphQL cost-weighted token deduction (pseudo-code)
function graphql_cost(document):
  cost = 0
  for each field in document:
    cost += COST_TABLE.get(field, 1)         # default: 1 point per field
    if field is a list:
      cost += estimated_items × child_cost   # pagination multiplier
  return cost

# Instead of deducting 1 token, deduct the query's cost
# Lua ARGV[1] becomes 'cost' instead of always '1'

The 429 contract: which limit tripped?

Returning a bare 429 is not enough when multiple limits coexist. A client needs to know three things: which scope rejected them, how long to wait, and what the state of the other limits is. This informs how they should throttle — a key-level rejection might mean only slowing that one integration thread, while an org-level rejection means all API traffic from the organization must pause until the daily counter resets at midnight.

By the numbers

The scenario

An organization has 20 apps, each configured for 100 req/s sustained and a per-key burst of 50 req/s. The org daily cap is 1,000,000 calls/day. All apps run their workers continuously at full tilt from 08:00 UTC.

The governing formula: which limit binds first?

At full sustained rate, the aggregate throughput across all apps is:

aggregate_rate  = num_apps × app_sustained_rate
                = 20 × 100 req/s
                = 2,000 req/s                    (modeled)

At that rate, the org's daily cap is exhausted in:

time_to_cap     = org_daily_cap / aggregate_rate
                = 1,000,000 / 2,000
                = 500 s ≈ 8 min 20 s              (modeled)

The daily cap is the binding constraint — it trips after about 8 minutes and 20 seconds of all apps running at full speed. From that point every request receives 429 scope: org and must wait until midnight UTC for the counter to reset.

Timestamped trace: which limit fires and when

Sizing the daily cap correctly

Work backwards from the business intent. If the goal is "20 apps can each run at full speed for an 8-hour business day," the needed daily cap is:

needed_cap = num_apps × app_rate × business_hours × 3600
           = 20 × 100 × 8 × 3600
           = 57,600,000 calls/day                (modeled)

At 1,000,000/day the org is undersized by a factor of 57. In practice, platforms set the daily cap at a level that prevents runaway loops and billing abuse, not at the theoretical maximum burst. They also tier it: a free org might get 10,000/day while an enterprise org gets 50,000,000/day.

Retry-After math for org-level rejections

retry_after = secs_until_midnight − secs_since_00:00_UTC
            = (24 × 3600) − (current_unix % 86400)
            = 86400 − (now % 86400)              (formula)

# At 08:20 UTC (30,000 s into the day):
retry_after = 86400 − 30000 = 56,400 s ≈ 15.7 hours

Trade-offs: design decisions you will face

How real platforms do it

Every major API platform publishes its rate limit model in its developer documentation. The patterns here are drawn from those public references.

Sources & further reading

• HubSpot — API usage details and rate limits
• GitHub — Rate limits for the REST API (primary + secondary limits)
• Shopify — API rate limits (leaky bucket + GraphQL query cost)
• Stripe — Rate limits documentation
• GitHub — GraphQL resource limitations and query cost model
• Redis — Lua scripting for atomic operations
• RFC 6585 §4 — HTTP 429 Too Many Requests