Retries, and why exactly-once delivery is a lie

Exactly-once delivery is impossible. Exactly-once processing is straightforward. The difference is where you put the responsibility: stop trying to make the network promise something it cannot, and make your handler not care how many times it is called.

Every queue, webhook provider and message broker that advertises exactly-once is describing the second thing.

Why the network cannot promise it

A sender delivers a message and waits for an acknowledgement. The acknowledgement does not arrive. The sender now knows exactly one thing: it did not get an ack.

It cannot distinguish between:

• the message never arrived
• the message arrived, was processed, and the ack was lost on the way back

Those two need opposite responses. Send again, and in the second case you have duplicated the work. Do not send, and in the first case you have lost the event. No amount of extra messages fixes this — the acknowledgement of the acknowledgement can also be lost. This is the two generals problem, and it is not an engineering gap; it is a proof.

So the only real choice is which failure you prefer:

Almost everything worth building picks at least once, because a duplicate is something you can defend against and a lost event is not.

The defence

Make processing idempotent — running it twice has the same effect as running it once. Then duplicates stop being a correctness problem and become a small waste of CPU.

Concretely, that is a unique key on something the sender controls:

INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;

If the RETURNING gives you a row, you are the first to see this event — do the work. If it gives you nothing, someone already did. This is the same pattern as webhook idempotency, and it generalises: the database decides who was first, because it is the only component that can.

For effects outside your database — charging a card, sending an email — pass an idempotency key to the downstream service and let it deduplicate too. Every serious payments API supports this precisely because their callers cannot promise exactly-once either.

At-least-once delivery plus an idempotent handler is what people mean by exactly-once. It is worth using the precise phrase — effectively once — because it names where the guarantee actually lives.

Backoff, and why jitter is not optional

Retrying immediately turns a brief outage into a sustained attack on your own service. Exponential backoff spaces attempts out:

const delay = Math.min(2 ** attempt * 100, 30_000)   // 200ms, 400ms, 800ms...

That is only half of it. If a hundred clients fail at the same instant — which is what an outage is — they all wait exactly 200 ms and all retry in the same millisecond. Your service comes back up, gets hit by the entire herd at once, falls over, and the cycle repeats with a tighter grip.

Randomise the wait:

const base = Math.min(2 ** attempt * 100, 30_000)
const delay = Math.random() * base          // full jitter

Simulating 200 clients that all fail at t=0, five attempts each:

no jitter     peak  200 requests in one 100ms window, spread over  5 windows
with jitter   peak  141 requests in one 100ms window, spread over 54 windows

Worth reading that honestly: jitter did not flatten the peak dramatically — 200 down to 141, about 30%. What it did was spread the same load across 54 windows instead of 5. The spikes stop being synchronised, which is what actually lets a recovering service stay up. Anyone promising you a tenfold drop in peak load from jitter alone has not measured it.

Know when to stop

Retrying forever is its own failure. A message that can never succeed — a malformed payload, a deleted record — will be retried until it is the only thing your workers are doing.

UPDATE jobs SET status = 'error', error = $2
 WHERE id = $1 AND attempts >= 5;

After a bounded number of attempts, move it aside. Whether you call that a dead-letter queue or a status column matters less than the two properties: it stops consuming capacity, and someone can see it. A dead-letter queue nobody looks at is a slower way of losing the event.

The counts that work in practice: 3–5 attempts for anything user-facing, more for background work that nobody is waiting on. Then alert on the size of the dead-letter set, not on individual failures.

Check it yourself

The herd effect, measured on your own machine:

// save as jitter.mjs
const CLIENTS = 200, ATTEMPTS = 5, BUCKET = 100

function run(jitter) {
  const hits = new Map()
  for (let c = 0; c < CLIENTS; c++) {
    let t = 0
    for (let a = 1; a <= ATTEMPTS; a++) {
      const base = 2 ** a * 100
      t += jitter ? Math.random() * base : base
      const b = Math.floor(t / BUCKET)
      hits.set(b, (hits.get(b) || 0) + 1)
    }
  }
  return { peak: Math.max(...hits.values()), buckets: hits.size }
}

for (const j of [false, true]) {
  const r = run(j)
  console.log(`${j ? 'with jitter' : 'no jitter  '}  peak ${r.peak}  windows ${r.buckets}`)
}

node jitter.mjs

Without jitter every client lands in the same five windows, forever, no matter how many clients there are. That is the number that matters: the peak does not improve as the outage lengthens, because everyone stays in lockstep.

Where this goes next

This is the contract behind every trigger in Workflow Builder: deliver at least once, deduplicate on arrival, back off with jitter, and dead-letter what cannot succeed.

Earlier in this series: receiving a webhook without losing events, verifying its signature, and the Postgres job queue that does the retrying.