Rate limits: respecting theirs, publishing yours

Read Retry-After and actually wait. A client that obeys the header finished 40 calls against a 10-per-second API in 3,015 ms having sent 43 requests. A client that retried immediately on every 429 finished the same work in 3,020 ms — after sending 55,930. Same throughput, 1,300x the traffic, and on a real API that second client is the one that gets banned.

The other half of the job is publishing your own limit so your callers can do the same thing. That is a token bucket and a 429 with a Retry-After header, and it is about ten lines.

The problem

You are pulling orders from a vendor API that allows 10 requests per second. Your worker has 40 to fetch. The naive loop looks fine in testing, because in testing you had four orders.

In production it hits the limit, gets a 429, and — because the retry logic was written for network errors — tries again immediately. That is not a retry. It is a busy loop with an HTTP request in it.

Read the headers, and read the units

Two families of headers exist, and they do not agree on what a number means.

The trap is that all three are integers, and a parser that does Number(header) * 1000 gets two of them catastrophically wrong:

// save as units.mjs — the same "wait this long" expressed three ways
const now = Date.now()
const cases = [
  ['Retry-After',       '120'],
  ['Retry-After',       new Date(now + 120_000).toUTCString()],
  ['RateLimit-Reset',   '120'],
  ['X-RateLimit-Reset', String(Math.floor(now / 1000) + 120)]
]

const naive   = v => Number(v) * 1000
const correct = v => {
  const n = Number(v)
  if (Number.isNaN(n)) return Date.parse(v) - Date.now()      // HTTP-date
  if (n > 1e9) return n * 1000 - Date.now()                   // epoch seconds, not a delta
  return n * 1000                                             // delta-seconds
}
const yrs = ms => (ms / 31_557_600_000).toFixed(1) + ' years'

for (const [h, v] of cases) {
  const a = naive(v), b = correct(v)
  console.log(
    `${h.padEnd(18)} ${String(v).padEnd(31)}` +
    ` naive=${(Number.isNaN(a) ? 'NaN -> no wait at all' : a > 1e10 ? yrs(a) : a + 'ms').padEnd(22)}` +
    ` correct=${Math.round(b / 1000)}s`)
}

Retry-After        120                             naive=120000ms               correct=120s
Retry-After        Sun, 30 Aug 2026 10:42:59 GMT   naive=NaN -> no wait at all  correct=120s
RateLimit-Reset    120                             naive=120000ms               correct=120s
X-RateLimit-Reset  1788086579                      naive=56.7 years             correct=120s

Two failure modes, both silent. NaN passed to setTimeout fires immediately, so a client that only ever saw the numeric form of Retry-After degenerates into the busy loop the moment a server sends the date form. And 56.7 years is a worker that never wakes up and never logs an error.

The rule: a value above 1e9 is a timestamp, not a duration. Any delta large enough to be confused with an epoch second is a delta you should not be sleeping through anyway.

Fixed window, sliding window, token bucket

A fixed window counts requests in the current clock second, minute or hour, then resets. It is the easiest to write and the one most people ship. Its flaw is at the boundary: a caller can spend the whole allowance at the end of one window and the whole allowance at the start of the next, and from the server's point of view nothing was violated.

Here it is against a limit of 10 per 1,000 ms — ten requests fired 150 ms before the boundary, ten more 20 ms after it:

// save as burst.mjs — 10 requests just before a window boundary, 10 just after
const sleep = ms => new Promise(r => setTimeout(r, ms))
const burst = async (path, n) =>
  (await Promise.all(Array.from({ length: n }, () =>
    fetch(`http://localhost:55541${path}`)))).filter(r => r.ok).length

for (const path of ['/fixed', '/sliding', '/bucket']) {
  await sleep(1000 - (Date.now() % 1000) + 850)   // land ~150ms before the boundary
  const t0 = Date.now()
  const a = await burst(path, 10)
  await sleep(1000 - (Date.now() % 1000) + 20)    // step over it
  const b = await burst(path, 10)
  console.log(`${path.padEnd(7)} before=${a} after=${b}  accepted=${a + b}` +
              ` in ${Date.now() - t0}ms  (limit is 10 per 1000ms)`)
}

/fixed  before=10 after=10  accepted=20 in 172ms  (limit is 10 per 1000ms)
/sliding before=10 after=0  accepted=10 in 175ms  (limit is 10 per 1000ms)
/bucket before=10 after=1  accepted=11 in 172ms  (limit is 10 per 1000ms)

Twenty requests in 172 ms, against a limit of ten per second. The capacity you have to provision for is double the number you published.

The sliding window log fixes it exactly — keep the timestamp of every request and count the ones inside the last second. It is also the only one of the three whose memory grows with traffic:

// save as mem.mjs — what each limiter costs to remember. node --expose-gc mem.mjs
const KEYS = 10_000, LIMIT = 1000
const mb = () => { global.gc(); global.gc(); return process.memoryUsage().heapUsed / 1e6 }
const hold = []

const base = mb()
const buckets = new Map()
for (let k = 0; k < KEYS; k++) buckets.set('bucket-key-' + k, { tokens: LIMIT, at: Date.now() })
hold.push(buckets)
const afterBucket = mb()

const logs = new Map()
for (let k = 0; k < KEYS; k++) logs.set('log-key-' + k, new Array(LIMIT).fill(Date.now()))
hold.push(logs)
const afterLog = mb()

console.log(`token bucket    ${(afterBucket - base).toFixed(1)} MB for ${KEYS} callers`)
console.log(`sliding log     ${(afterLog - afterBucket).toFixed(1)} MB for ${KEYS} callers at ${LIMIT} req/window`)

token bucket    1.6 MB for 10000 callers
sliding log     81.3 MB for 10000 callers at 1000 req/window

The token bucket is the compromise, and it is what you want. Each caller has a bucket that holds LIMIT tokens and refills at LIMIT/WINDOW tokens per millisecond. A request costs a token. There are no windows, so there is no boundary to exploit — the 11th request in the demo above got through because 120 ms of refill had genuinely happened, which is the correct answer.

The fix: a token bucket, and the headers that go with it

No timers, no background job. The refill is arithmetic done at read time. This is the whole server the numbers above came from — Node 23, no dependencies:

// save as server.mjs  —  three limiters, same limit, one endpoint each
import { createServer } from 'node:http'

const LIMIT = 10          // requests
const WINDOW = 1000       // per second

// --- fixed window -------------------------------------------------------
const windows = new Map()
function fixedWindow(key) {
  const w = Math.floor(Date.now() / WINDOW)
  const id = key + ':' + w
  const used = (windows.get(id) || 0) + 1
  windows.set(id, used)
  return { ok: used <= LIMIT, remaining: Math.max(0, LIMIT - used),
           resetMs: (w + 1) * WINDOW - Date.now() }
}

// --- sliding window log -------------------------------------------------
const logs = new Map()
function slidingWindow(key) {
  const now = Date.now()
  const hits = (logs.get(key) || []).filter(t => t > now - WINDOW)
  const ok = hits.length < LIMIT
  if (ok) hits.push(now)
  logs.set(key, hits)
  return { ok, remaining: Math.max(0, LIMIT - hits.length),
           resetMs: hits.length ? hits[0] + WINDOW - now : 0 }
}

// --- token bucket -------------------------------------------------------
const buckets = new Map()
function tokenBucket(key, cost = 1) {
  const now = Date.now()
  const b = buckets.get(key) || { tokens: LIMIT, at: now }
  b.tokens = Math.min(LIMIT, b.tokens + (now - b.at) * (LIMIT / WINDOW))
  b.at = now
  const ok = b.tokens >= cost
  if (ok) b.tokens -= cost
  buckets.set(key, b)
  const deficit = ok ? 0 : cost - b.tokens
  return { ok, remaining: Math.floor(b.tokens), resetMs: Math.ceil(deficit * (WINDOW / LIMIT)) }
}

createServer((req, res) => {
  const key = req.headers['x-api-key'] || 'anonymous'
  const r = req.url.startsWith('/fixed')   ? fixedWindow(key)
          : req.url.startsWith('/sliding') ? slidingWindow(key)
          : req.url.startsWith('/global')  ? tokenBucket('everyone')  // one bucket for all callers
          :                                  tokenBucket(key)         // one bucket per caller
  const resetSec = Math.ceil(r.resetMs / 1000)
  res.setHeader('RateLimit-Policy', `"burst";q=${LIMIT};w=1`)
  res.setHeader('RateLimit', `"burst";r=${r.remaining};t=${resetSec}`)
  res.setHeader('RateLimit-Limit', LIMIT)
  res.setHeader('RateLimit-Remaining', r.remaining)
  res.setHeader('RateLimit-Reset', resetSec)          // delta-seconds, NOT a timestamp
  if (!r.ok) {
    res.setHeader('Retry-After', resetSec)
    res.writeHead(429).end('slow down\n')
    return
  }
  res.writeHead(200).end('ok\n')
}).listen(55541, () => console.log('listening on 55541'))

The cost parameter is the part people leave out and regret. A search endpoint that fans out to three services is not one unit of load, and pricing it as one means your published limit describes nothing. Charge expensive endpoints more tokens rather than giving them a separate limit nobody can reason about.

Publishing yours

Note where the headers are set: on every response, not only on the 429. A client cannot slow down gracefully if the only signal it gets is the wall it already hit.

Both header shapes are there on purpose. The IETF draft has been through two: three separate RateLimit-* fields, and a newer consolidated RateLimit plus RateLimit-Policy structured field. Emitting both costs about 60 bytes per response and means you do not have to care which one your caller's HTTP library learned. A real 429 from that server:

HTTP/1.1 429 Too Many Requests
RateLimit-Policy: "burst";q=10;w=1
RateLimit: "burst";r=0;t=1
RateLimit-Limit: 10
RateLimit-Remaining: 0
RateLimit-Reset: 1
Retry-After: 1

Use 429 (RFC 6585), not 503, and not 403. A caller can write one branch for "you are going too fast"; it cannot write a branch for "something was wrong, possibly your credentials".

One thing measurement changed about our own advice: we expected a client that obeys Retry-After to be measurably slower, because the header's resolution is whole seconds and a 100 ms deficit gets rounded up to a full one. It is not. Across five runs the obedient client came in at 3,015–3,065 ms and the blind one at 3,018–3,820 ms, sending between 47,450 and 75,021 requests — the blind client was sometimes slower, because fifty thousand rejected requests are still fifty thousand requests the server has to parse. Obeying the header costs nothing. That is worth knowing, because "but backing off makes us slow" is the argument that keeps the busy loop in the codebase.

One number for everyone is the wrong number

A single global limiter does not protect your service from a customer. It protects your service by penalising every other customer. One caller sending 40 requests, two others sending 3 each, in the same second:

// save as neighbour.mjs — one noisy caller, two quiet ones
const call = (path, key) => fetch(`http://localhost:55541${path}`, { headers: { 'x-api-key': key } })

async function round(path) {
  const jobs = []
  for (let i = 0; i < 40; i++) jobs.push(['noisy', call(path, 'acme')])
  for (let i = 0; i < 3; i++)  jobs.push(['quiet-a', call(path, 'bakery')])
  for (let i = 0; i < 3; i++)  jobs.push(['quiet-b', call(path, 'clinic')])
  const out = {}
  for (const [who, p] of jobs) {
    const r = await p
    out[who] ??= { ok: 0, blocked: 0 }
    r.ok ? out[who].ok++ : out[who].blocked++
  }
  console.log(path.padEnd(8), Object.entries(out).map(([k, v]) => `${k}: ${v.ok} ok / ${v.blocked} 429`).join('   '))
}

await round('/global')                                  // one bucket for everyone
await new Promise(r => setTimeout(r, 2000))
await round('/bucket')                                  // one bucket per API key

/global  noisy: 10 ok / 30 429   quiet-a: 0 ok / 3 429   quiet-b: 0 ok / 3 429
/bucket  noisy: 10 ok / 30 429   quiet-a: 3 ok / 0 429   quiet-b: 3 ok / 0 429

The noisy caller is throttled identically either way. The difference is entirely in what happens to everyone else: under one global bucket the two small customers lose every request they made, and the only thing they did wrong was share a server with somebody busy.

Key the bucket on whatever identifies the payer — API key, tenant id, account — not on IP. IP buckets punish everyone behind one office NAT and do nothing about a caller with a hundred IPs. Keep a global bucket too if you like, but as a circuit breaker sized well above the sum of the plans, not as the primary control.

Check it yourself

Node 23, no dependencies, nothing leaves the machine. With server.mjs above saved, add the client:

// save as obey.mjs — 40 calls to make, three client strategies
const sleep = ms => new Promise(r => setTimeout(r, ms))
const WORK = 40

async function drive(name, strategy) {
  let sent = 0, rejected = 0, done = 0
  const bucket = { tokens: 10, at: Date.now() }
  const t0 = Date.now()
  while (done < WORK) {
    if (strategy === 'proactive') {              // client-side copy of the published limit
      const now = Date.now()
      bucket.tokens = Math.min(10, bucket.tokens + (now - bucket.at) * 0.01)
      bucket.at = now
      if (bucket.tokens < 1) { await sleep(Math.ceil((1 - bucket.tokens) * 100)); continue }
      bucket.tokens -= 1
    }
    const res = await fetch('http://localhost:55541/bucket'); sent++
    if (res.status === 429) {
      rejected++
      if (strategy === 'obedient') await sleep(Number(res.headers.get('retry-after')) * 1000)
      continue                                    // 'blind' retries with no wait at all
    }
    done++
  }
  console.log(`${name.padEnd(10)} sent=${String(sent).padStart(5)}  429s=${String(rejected).padStart(5)}  wall=${Date.now() - t0}ms`)
}

await drive('blind', 'blind')
await sleep(1500)
await drive('obedient', 'obedient')
await sleep(1500)
await drive('proactive', 'proactive')

node server.mjs &
node obey.mjs

blind      sent=55930  429s=55890  wall=3020ms
obedient   sent=   43  429s=    3  wall=3015ms
proactive  sent=   41  429s=    1  wall=3103ms

The blind line is the only one that moves between runs — it is bounded by how fast your loopback is, not by anything it achieves.

Note the last line. The proactive client mirrors the published limit locally and still collected a 429 — sometimes one, sometimes zero. It always will, because the server's clock and your clock are not the same clock and the network sits between them. A client-side limiter reduces 429s; it never removes the need to handle one. Keep the Retry-After branch.

The code

Runnable, and CI keeps it that way: CSTSolution/examples/rate-limiter — all three limiters, with a test that fails if the fixed window stops letting 2x through.

git clone https://github.com/CSTSolution/examples
cd examples/rate-limiter

Where this goes next

The wait itself needs jitter once more than one worker is involved — a hundred clients that all read Retry-After: 1 all wake in the same millisecond, which is the thundering herd, measured. And a request that has to wait 30 seconds should not be holding a worker: park it in a queue and let something else run, which is the difference between cron, a queue and a stream.

If you are on the receiving end, the same bucket belongs in front of your webhook endpoint — receiving a webhook without losing events covers the ack-fast, work-later shape that lets you return 200 instead of 429 under a burst.

Every connector in Workflow Builder carries the vendor's published limit as configuration and a bucket per credential, because two customers of ours sharing one vendor account is the same noisy-neighbour problem one level up.