Polling an API that has no webhooks, without hammering it

Send If-None-Match with every poll, page with a composite (updated_at, id) cursor, and never let a poll start while the last one is still running. Sixty polls of an unchanged collection cost 60 units of quota and 38,880 bytes without conditional requests, and 1 unit and 13,038 bytes with them. The cursor is not an optimisation: paging on updated_at alone silently loses records, and this article reproduces it in twenty lines.

Polling is what you do when the vendor has no webhooks. It is not a worse integration — it is a different one, with three failure modes that all look like success.

The problem

A vendor API. 5,000 calls an hour, no webhooks, no streaming. You need new and changed orders within about a minute.

The obvious loop is setInterval, fetch /orders, diff against what you have. It works on day one. What it does on day ninety is burn your entire quota on responses that are byte-identical to the last one, hold a worker open while a slow call finishes, and — the expensive one — miss records without ever raising an error.

Conditional requests, and what a 304 costs

An ETag is an opaque version string for a response. Send it back as If-None-Match and the server answers 304 Not Modified with no body.

The part that makes this worth doing rather than merely tidy: on several major APIs a 304 does not count against your rate limit. GitHub documents exactly this for its REST API. Not every vendor does it, so check theirs — but where it holds, the cost of polling drops to almost nothing.

Sixty polls of a collection that never changed, with and without the header:

no conditionals  calls=60  quota_used=60  304s=0   bytes=38880  bodies_parsed=60
if-none-match    calls=60  quota_used=1   304s=59  bytes=13038  bodies_parsed=1

Same sixty HTTP requests. One unit of quota instead of sixty, and 13,038 bytes instead of 38,880 — a 200 here is 648 bytes on the wire and a 304 is 210, headers and all. The bodies_parsed column is the one people forget: 59 JSON parses and 59 diffs against your database that you no longer do.

The client side is four lines:

let etag = null
const res = await fetch(url, etag ? { headers: { 'if-none-match': etag } } : {})
if (res.status === 304) return []          // nothing changed, and it was free
etag = res.headers.get('etag')

Last-Modified and If-Modified-Since are the older pair and work the same way, with one flaw that matters.

Why If-Modified-Since is the weaker of the two

An HTTP-date has one-second resolution. A record that changes in the same second as the one you were told about is, as far as If-Modified-Since can express, not a change at all.

Here a seventh order arrives 500 ms after the last one, inside the same second:

// save as granularity.mjs — a change 500ms after the last one, seen by both validators
const API = 'http://localhost:55542'
await fetch(API + '/untouch')

const first = await fetch(API + '/orders')
const etag = first.headers.get('etag'), lastModified = first.headers.get('last-modified')
console.log(`before   rows=${(await first.json()).data.length}  Last-Modified: ${lastModified}`)

await fetch(API + '/touch')                       // order 7 lands at 10:00:03.500Z

const ims = await fetch(API + '/orders', { headers: { 'if-modified-since': lastModified } })
const inm = await fetch(API + '/orders', { headers: { 'if-none-match': etag } })
console.log(`after    If-Modified-Since -> ${ims.status}   If-None-Match -> ${inm.status}`)
console.log(`         Last-Modified is still ${inm.headers.get('last-modified')}` +
            `, rows now = ${(await inm.json()).data.length}`)

before   rows=6  Last-Modified: Sun, 30 Aug 2026 10:00:03 GMT
after    If-Modified-Since -> 304   If-None-Match -> 200
         Last-Modified is still Sun, 30 Aug 2026 10:00:03 GMT, rows now = 7

The row count went from six to seven and If-Modified-Since reported no change — correctly, by the letter of the spec, because the Last-Modified value did not change either. Send both if the server offers both, but treat the ETag as the authority. If a vendor only offers Last-Modified, assume one second of blindness and re-check the boundary rather than trusting a 304 at the edge.

Cursors, and the boundary that eats records

Conditional requests tell you whether something changed. They do not tell you what. For that you want updated_since or a cursor, so you fetch the delta instead of re-reading the collection.

The natural implementation is: ask for everything after the last timestamp you saw, remember the last row's updated_at, repeat. It is wrong, and the way it is wrong is the reason this section exists.

Six orders. Three of them — 3, 4 and 5 — share the timestamp 10:00:02, because they were written by one transaction or one bulk import. Page size 2.

Order 5 is at 10:00:02. The filter is updated_at > 10:00:02. Order 5 is never returned, never will be, and nothing anywhere reports an error.

The instinct is to change > to >= and deduplicate on id. That is worse:

gt      requests=4  got=[1,2,3,4,6]  missing=[5]
  gte: page repeated with no new rows — this loop does not terminate
gte     requests=4  got=[1,2,3,4]  missing=[5,6]
keyset  requests=4  got=[1,2,3,4,5,6]  missing=[]

With >=, once the cursor lands on 10:00:02 the next page is orders 3 and 4 again — the same page, forever, because the cursor cannot advance past a timestamp that fills a whole page. A poller that pages by >= with a page size smaller than the largest group of same-timestamp rows does not just re-read; it stops making progress entirely. That loop above only ended because the demo detects the repeat and bails.

The fix is a keyset cursor: order by (updated_at, id) and ask for rows strictly after the last pair, not after the last timestamp.

SELECT * FROM orders
 WHERE (updated_at, id) > ($1, $2)
 ORDER BY updated_at, id
 LIMIT 50;

Postgres compares row constructors left to right, so that one clause means "later timestamp, or same timestamp and higher id". Against an API you do not control, send both parts — most updated_since endpoints also accept an after_id or an opaque cursor, and if the vendor offers an opaque cursor, use it: it is the vendor promising to solve this problem for you, and it is the only version that survives them changing their sort order.

If the vendor offers neither, you are stuck overlapping: request from last_seen_at minus a safety margin, and deduplicate by id on your side. Pick the margin from the vendor's own clock skew, not from optimism. This is the same idempotency key you need for webhooks, doing the same job.

Poll faster when things are happening

A fixed interval is a bet that the world changes at a constant rate. It does not. Back off on empty polls and reset on a non-empty one:

interval = data.length ? MIN : Math.min(interval * 2, MAX)

Twelve seconds with writes at 0–2s and 9–11s and nothing in between, against a 500 ms floor and an 8 s ceiling:

fixed          calls= 24  quota_used= 24  events=10  mean_latency=234ms  max_latency=463ms
adaptive       calls= 12  quota_used= 12  events=10  mean_latency=414ms  max_latency=1003ms
adaptive+etag  calls= 12  quota_used= 10  events=10  mean_latency=413ms  max_latency=1004ms

Half the calls. It is a real trade and worth stating plainly: mean detection latency went from about 235 ms to about 415 ms, and worst case from under 500 ms to just over a second. Adaptive polling is not free — it buys quota with latency, and the exchange rate is set by your ceiling. Cap the ceiling at the freshness you actually promised, not at the largest number that still feels responsive.

The third line is the correction. We expected conditional requests to help here as much as they did on the full collection, and they did not: 12 calls became 10 units of quota, not 1. If-None-Match pays off on an endpoint that returns the same big body over and over. On a cursor endpoint the body is already {"data":[]} — there is almost nothing left to save, and the only 304s you get are the repeated empty polls. Use both, but expect the cursor to be doing the work.

The overlap nobody notices until the upstream slows down

setInterval(poll, 400) does not mean "poll every 400 ms". It means "start a poll every 400 ms", and if the API takes 900 ms you now have polls running concurrently, all of which read the cursor before any of them wrote it.

Four seconds of a 400 ms schedule against a 900 ms call, with a producer writing a new order every 300 ms:

interval  processed=30  distinct=11  duplicates=19  max_concurrent_polls=3
guarded   processed=9  distinct=9  duplicates=0  max_concurrent_polls=1
serial    processed=13  distinct=13  duplicates=0  max_concurrent_polls=1

Three polls in flight at once, and 19 of 30 processed records were duplicates — every overlapping poll re-fetched the window its predecessor had not yet acknowledged. On a poller whose handler charges a card or sends an email, that is not wasted CPU.

Two fixes, and they are not equivalent. guarded keeps the timer and drops any tick that arrives while a poll is in flight. serial waits for the poll to finish and then sleeps for the interval. Both eliminate duplicates; the serial loop got through 12 or 13 records to the guarded loop's 9, because a dropped tick is still a poll that did not happen. Prefer the serial loop. Keep the guard as well if anything else can trigger a poll — a manual "sync now" button is a second caller into the same cursor.

If the poll itself is slow enough that this is a real constraint, the poll should not be doing the work. Have it enqueue and return, which is the difference between cron, a queue and an event stream, and let a queue own the retries.

Check it yourself

Node 23, no dependencies, nothing leaves the machine. The API below has no webhooks, a 5,000-call quota, ETags, and three cursor semantics so you can watch two of them lose data.

// save as api.mjs — a small API with no webhooks: 5000 calls/hour, ETag, cursor paging
import { createServer } from 'node:http'
import { createHash } from 'node:crypto'

const T = s => new Date(Date.parse('2026-08-30T10:00:00Z') + s * 1000).toISOString()
// Three of these six orders share one timestamp. That is not contrived — it is
// what a bulk import, or one transaction touching three rows, looks like.
const orders = [
  { id: 1, ref: 'A-1', updated_at: T(0) },
  { id: 2, ref: 'A-2', updated_at: T(1) },
  { id: 3, ref: 'A-3', updated_at: T(2) },
  { id: 4, ref: 'A-4', updated_at: T(2) },
  { id: 5, ref: 'A-5', updated_at: T(2) },
  { id: 6, ref: 'A-6', updated_at: T(3) }
]
let quota = 5000, calls = 0, notModified = 0, bytes = 0, nextId = 100

const serve = (req, res) => {
  const url = new URL(req.url, 'http://x')
  const p = url.pathname
  if (p === '/stats') return res.writeHead(200, { 'content-type': 'application/json' })
    .end(JSON.stringify({ calls, quota_used: 5000 - quota, not_modified: notModified, bytes }))
  if (p === '/reset')   { quota = 5000; calls = notModified = bytes = 0; return res.end('ok') }
  if (p === '/append')  { orders.push({ id: nextId, ref: 'B-' + nextId++, updated_at: new Date().toISOString() }); return res.end('ok') }
  if (p === '/touch')   { orders.push({ id: 7, ref: 'A-7', updated_at: T(3.5) }); return res.end('ok') }
  if (p === '/untouch') { if (orders.length > 6) orders.pop(); return res.end('ok') }

  const before = req.socket.bytesWritten
  res.on('finish', () => { bytes += req.socket.bytesWritten - before })
  calls++

  const since = url.searchParams.get('updated_since')
  const afterId = Number(url.searchParams.get('after_id') || 0)
  const limit = Number(url.searchParams.get('limit') || 100)
  const op = url.searchParams.get('op') || 'gt'

  let rows = orders
  if (since) rows = orders.filter(o =>
      op === 'gte'    ? o.updated_at >= since
    : op === 'keyset' ? o.updated_at > since || (o.updated_at === since && o.id > afterId)
    :                   o.updated_at > since)
  rows = rows.slice(0, limit)

  const body = JSON.stringify({ data: rows })
  const etag = '"' + createHash('sha256').update(body).digest('hex').slice(0, 16) + '"'
  const lastModified = new Date(Math.max(...orders.map(o => Date.parse(o.updated_at)))).toUTCString()

  const finish = () => {
    res.setHeader('ETag', etag)
    res.setHeader('Last-Modified', lastModified)
    res.setHeader('Cache-Control', 'no-cache')
    const fresh = req.headers['if-none-match'] === etag ||
      (req.headers['if-modified-since'] &&
       Date.parse(req.headers['if-modified-since']) >= Date.parse(lastModified))
    if (fresh) { notModified++; return res.writeHead(304).end() }   // a 304 costs no quota
    quota--
    res.writeHead(200, { 'content-type': 'application/json' }).end(body)
  }

  const delay = Number(url.searchParams.get('delay') || 0)   // pretend the upstream is slow
  delay ? setTimeout(finish, delay) : finish()
}

createServer(serve).listen(55542, () => console.log('api on 55542'))

The cursor demo — this is the one to run first:

// save as cursor.mjs — three ways to page by updated_at. Two of them are wrong.
const PAGE = 2, MAX_REQ = 8

async function drain(op) {
  let since = '2026-08-29T00:00:00.000Z', afterId = 0
  const seen = [], ids = new Set()
  let requests = 0
  while (requests < MAX_REQ) {
    const u = `http://localhost:55542/orders?op=${op}&limit=${PAGE}` +
              `&updated_since=${encodeURIComponent(since)}&after_id=${afterId}`
    const { data } = await (await fetch(u)).json()
    requests++
    if (!data.length) break
    let progress = false
    for (const o of data) if (!ids.has(o.id)) { ids.add(o.id); seen.push(o.id); progress = true }
    const last = data[data.length - 1]
    since = last.updated_at
    afterId = last.id
    if (op === 'gte' && !progress) {
      console.log(`  ${op}: page repeated with no new rows — this loop does not terminate`); break
    }
  }
  const missing = [1, 2, 3, 4, 5, 6].filter(i => !ids.has(i))
  console.log(`${op.padEnd(7)} requests=${requests}  got=[${seen}]  missing=[${missing}]`)
}

for (const op of ['gt', 'gte', 'keyset']) await drain(op)

node api.mjs &
node cursor.mjs

gt      requests=4  got=[1,2,3,4,6]  missing=[5]
  gte: page repeated with no new rows — this loop does not terminate
gte     requests=4  got=[1,2,3,4]  missing=[5,6]
keyset  requests=4  got=[1,2,3,4,5,6]  missing=[]

Order 5 exists, was never deleted, and your poller will never see it. That is the argument for the composite cursor in one line of output.

Then the conditional-request and overlap demos, against the same server:

// save as conditional.mjs — 60 polls of a collection that changed once
const API = 'http://localhost:55542'

async function poll(n, conditional) {
  await fetch(API + '/reset')
  let etag = null, changed = 0
  for (let i = 0; i < n; i++) {
    const headers = conditional && etag ? { 'if-none-match': etag } : {}
    const res = await fetch(API + '/orders', { headers })
    if (res.status === 304) continue
    etag = res.headers.get('etag')
    await res.json()
    changed++
  }
  const s = await (await fetch(API + '/stats')).json()
  console.log(`${(conditional ? 'if-none-match' : 'no conditionals').padEnd(16)}` +
    ` calls=${s.calls}  quota_used=${s.quota_used}  304s=${s.not_modified}` +
    `  bytes=${s.bytes}  bodies_parsed=${changed}`)
}

await poll(60, false)
await poll(60, true)

// save as overlap.mjs — a 400ms poll schedule against a 900ms API call
const API = 'http://localhost:55542'
const sleep = ms => new Promise(r => setTimeout(r, ms))

async function measure(mode) {
  await fetch(API + '/reset')
  let cursor = new Date().toISOString(), cursorId = 0, running = false, stop = false
  const processed = []
  let inFlight = 0, maxInFlight = 0

  async function pollOnce() {
    if (mode === 'guarded' && running) return       // skip the tick, the last poll is still out
    running = true
    maxInFlight = Math.max(maxInFlight, ++inFlight)
    const from = cursor, fromId = cursorId          // cursor read here...
    const u = `${API}/orders?op=keyset&delay=900&limit=50` +
              `&updated_since=${encodeURIComponent(from)}&after_id=${fromId}`
    const { data } = await (await fetch(u)).json()
    for (const o of data) processed.push(o.id)
    if (data.length) { cursor = data.at(-1).updated_at; cursorId = data.at(-1).id }
    inFlight--; running = false                     // ...and written here, 900ms later
  }

  const producer = setInterval(() => fetch(API + '/append'), 300)
  const timer = mode === 'serial' ? null : setInterval(pollOnce, 400)
  if (mode === 'serial') (async () => { while (!stop) { await pollOnce(); await sleep(400) } })()

  await sleep(4000)
  stop = true; clearInterval(producer); if (timer) clearInterval(timer)
  await sleep(1200)

  const distinct = new Set(processed).size
  console.log(`${mode.padEnd(9)} processed=${processed.length}  distinct=${distinct}` +
              `  duplicates=${processed.length - distinct}  max_concurrent_polls=${maxInFlight}`)
}

for (const m of ['interval', 'guarded', 'serial']) await measure(m)

And the adaptive interval, which needs a producer that goes quiet:

// save as adaptive.mjs — 12 seconds: busy, then quiet, then busy again
const API = 'http://localhost:55542'
const sleep = ms => new Promise(r => setTimeout(r, ms))

const script = []                                   // writes at 0-2s and 9-11s
for (let t = 0; t < 2000; t += 400) script.push(t)
for (let t = 9000; t < 11000; t += 400) script.push(t)

async function run(mode) {
  await fetch(API + '/reset')
  let cursor = new Date().toISOString(), cursorId = 0
  let interval = 500, stop = false, etag = null
  const lat = []
  for (const at of script) setTimeout(() => fetch(API + '/append'), at)

  const loop = (async () => {
    while (!stop) {
      const u = `${API}/orders?op=keyset&limit=50` +
                `&updated_since=${encodeURIComponent(cursor)}&after_id=${cursorId}`
      const res = await fetch(u, mode === 'adaptive+etag' && etag ? { headers: { 'if-none-match': etag } } : {})
      const data = res.status === 304 ? [] : (await res.json()).data
      if (res.status !== 304) etag = res.headers.get('etag')
      const now = Date.now()
      for (const o of data) lat.push(now - Date.parse(o.updated_at))
      if (data.length) { cursor = data.at(-1).updated_at; cursorId = data.at(-1).id }
      if (mode.startsWith('adaptive')) interval = data.length ? 500 : Math.min(interval * 2, 8000)
      await sleep(interval)
    }
  })()
  await sleep(12_000); stop = true; await loop

  const s = await (await fetch(API + '/stats')).json()
  const mean = Math.round(lat.reduce((a, b) => a + b, 0) / lat.length)
  console.log(`${mode.padEnd(14)} calls=${String(s.calls).padStart(3)}` +
              `  quota_used=${String(s.quota_used).padStart(3)}  events=${lat.length}` +
              `  mean_latency=${mean}ms  max_latency=${Math.max(...lat)}ms`)
}

for (const m of ['fixed', 'adaptive', 'adaptive+etag']) await run(m)

Restart api.mjs between demos — overlap.mjs and adaptive.mjs both append rows, so the row counts in the earlier runs assume a fresh server.

Where this goes next

The order of operations, once all three are in:

1. Conditional request on the cursor endpoint. Free when nothing changed, and on some APIs free of quota too.
2. Keyset cursor on (updated_at, id), or the vendor's opaque cursor if they have one. Never the bare timestamp.
3. Serial loop, adaptive interval, ceiling set to the freshness you promised.
4. Enqueue, do not process. The poll's only job is to notice.

Step 4 is where polling stops being special. Once the poller writes rows into a queue, everything downstream is the same machinery a webhook would have fed: at-least-once delivery and an idempotent handler, a worker pulling with SKIP LOCKED, and a dead-letter status somebody looks at. And when you are the one being polled, the other side of this is publishing a rate limit your callers can obey.

Workflow Builder runs the polling triggers this way for every connector whose vendor has no webhooks — one cursor per connection, stored, and a run row for each poll that found something.