How to receive a webhook without losing events

Accept the request, write the event down, return 200, and do the actual work afterwards. A webhook endpoint that processes inline looks fine in development and loses events in production, because the sender's patience is shorter than your handler.

Everything else in this article is a consequence of that one rule.

What you are actually agreeing to

A webhook is someone else's server calling yours when something happens. You give them a URL; they POST to it. That is the whole protocol, and it hides three commitments people do not realise they are making:

• You must answer quickly. Most providers time out in 5–30 seconds. Some are stricter.
• You will receive duplicates. A timeout on their side means they resend. So does a network blip. So does their own retry logic.
• A non-2xx means "try again". Until it means "give up", and then the event is gone forever.

Answer fast, work later

app.post('/webhooks/acme', express.json(), async (req, res) => {
  const event = req.body

  // 1. Write it down. Nothing clever, just persist it.
  await db.query(
    `INSERT INTO webhook_events (id, source, payload)
     VALUES ($1, 'acme', $2) ON CONFLICT (id) DO NOTHING`,
    [event.id, event],
  )

  // 2. Acknowledge. We have the event; their job is done.
  res.status(200).end()

  // 3. Do the work somewhere else, on a worker.
})

Three lines of real work. The handler does not call your billing provider, does not send an email, does not ask a model anything. It writes a row and returns.

Why this matters more than it looks: if your handler takes eight seconds because something downstream is slow, the sender times out, marks the delivery failed and sends it again — while your first handler is still running. Now you have two handlers processing the same event concurrently, and a slow day has become a double-charged customer.

The work itself belongs on a queue. A Postgres table and SKIP LOCKED is enough; you already have the database.

Duplicates are normal, so make them harmless

Every serious provider sends an event id. Use it as a primary key and let the database decide who was first:

CREATE TABLE webhook_events (
  id         TEXT PRIMARY KEY,          -- THEIR id, not one you generate
  source     TEXT NOT NULL,
  payload    JSONB NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  processed_at TIMESTAMPTZ
);

INSERT INTO webhook_events (id, source, payload)
VALUES ($1, $2, $3)
ON CONFLICT (id) DO NOTHING;

ON CONFLICT DO NOTHING is doing real work here. Two concurrent deliveries of the same event race to insert; exactly one wins, and it is the database deciding — the only component in the system that can decide it correctly. Checking "does this exist?" in application code before inserting has a gap between the check and the write, and that gap is where the double charge lives.

If the provider does not send an id, build one from the payload:

const id = crypto.createHash('sha256')
  .update(`${source}:${rawBody}`).digest('hex')

Not perfect — two genuinely identical events become one — but for most payloads a timestamp inside makes them unique anyway, and it is far better than processing everything twice.

What your status code actually means

This is the part that quietly loses events.

So a bug that throws a 500 means the provider keeps retrying — good, until it exhausts its schedule (often hours or days) and drops the event permanently. Whereas a 400 because you could not parse one field can make some providers give up immediately.

The safe pattern: return 200 for anything you have successfully stored, even if you cannot process it yet. Storing it means you own it, and you can retry on your own terms rather than theirs.

try {
  await store(event)
  res.status(200).end()
} catch (err) {
  // We genuinely could not persist it. Ask them to retry.
  req.log.error({ err }, 'failed to store webhook')
  res.status(500).end()
}

Reserve 5xx for "I could not write it down". That is the only failure where their retry actually helps.

Testing without a public URL

You do not need a tunnel to build this. A webhook is an HTTP POST, so send one:

curl -X POST localhost:3000/webhooks/acme \
  -H 'content-type: application/json' \
  -d '{"id":"evt_8f2a","type":"invoice.paid","amount":4200}'

A tunnel is only needed when you want their server to reach your laptop. For building the handler, and for every test in CI, curl is the whole story — and unlike a tunnel it works offline and does not expire.

Check it yourself

Prove the idempotency, which is the part that is easy to get subtly wrong:

docker run --rm -d --name wh -e POSTGRES_PASSWORD=demo -p 55433:5432 postgres:16
until docker exec wh pg_isready -U postgres >/dev/null 2>&1; do sleep 1; done

DB=postgresql://postgres:demo@localhost:55433/postgres
psql $DB -q -c "CREATE TABLE webhook_events (id TEXT PRIMARY KEY, payload JSONB);"

# the same event, delivered three times, concurrently
for i in 1 2 3; do
  psql $DB -q -c "INSERT INTO webhook_events (id, payload)
                  VALUES ('evt_8f2a', '{\"n\":$i}') ON CONFLICT (id) DO NOTHING;" &
done
wait

psql $DB -c "SELECT count(*) FROM webhook_events;"

count is 1. Remove ON CONFLICT DO NOTHING and run it again: two of the three fail loudly with a duplicate-key error, which is at least honest — but if you had written a SELECT check instead of a constraint, all three would have inserted and you would never have known.

docker rm -f wh

The code

Runnable, and CI keeps it that way: CSTSolution/examples/webhook-receiver — a receiver with signature verification, replay rejection and idempotency.

git clone https://github.com/CSTSolution/examples
cd examples/webhook-receiver

Where this goes next

Next in this series: verifying the signature, because so far anyone who learns your URL can post to it. That is one HMAC comparison and three details that decide whether it is worth anything.

This shape — accept, store, enqueue, process — is what Workflow Builder does with every incoming trigger, and how Simple CRM takes events from the tools it connects to.