You Received the Same Webhook Twice. That's the Contract, Not a Bug.

Someone got charged once and emailed twice. Or an order shipped twice. You dig into the logs and find the same event.id processed at 14:02:07 and again at 14:02:41, and the first instinct is that the provider is broken.

The provider is not broken. It is doing exactly what it promises. Stripe, GitHub, Shopify, Twilio, and every other webhook sender of consequence guarantee at-least-once delivery, and at-least-once is a precise term of art: it means duplicates are part of the contract. Building as though delivery were exactly-once is the actual bug, and it is one that only surfaces in production, usually during your busiest hour.

TL;DR

• At-least-once means duplicates are guaranteed to happen eventually, not rarely
• Exactly-once delivery is not purchasable at any price; exactly-once processing is, and you build it on your side
• A unique index on the provider's event ID handles 90% of it in one line of SQL
• Non-transactional side effects (emails, third-party APIs) need a claim-then-commit pattern
• Out-of-order arrival is the sibling problem nobody plans for, and it needs versioning, not deduplication

Why nobody sells exactly-once

Picture the sender's position. It POSTs your event, and the TCP connection dies before the response comes back. Your server may have processed it fully and the ACK got lost, or your server may have died before touching it. From the outside, those two states are identical.

The sender has exactly two options:

Retry. If you already processed it, you now have a duplicate. Don't retry. If you never processed it, the event is lost forever.

That's the whole dilemma, and there is no third door. Providers universally pick retry, because a duplicate payment notification is an annoyance you can engineer around and a missing payment notification is silent revenue loss you cannot detect. They made the right call. It just means the deduplication has to live somewhere, and the only place with enough context is your database.

Real-world sources of the duplicate, roughly in order of how often we see them:

• Your response was slow and the sender timed out, but your handler completed anyway
• A load balancer or proxy in front of your app returned 502 after your app returned 200
• Your handler succeeded, then threw during cleanup, and your framework turned that into a 500
• A deploy killed the process after the write but before the response
• You clicked "resend" in a dashboard and forgot

Note that four of those five are cases where the work was done and the acknowledgement was lost. That's the normal case, not an exotic one.

Pattern 1: Unique index on the event ID

If you do one thing, do this.

CREATE TABLE processed_events (
  event_id    TEXT PRIMARY KEY,
  provider    TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

const { rowCount } = await db.query(
  `INSERT INTO processed_events (event_id, provider)
   VALUES ($1, $2) ON CONFLICT (event_id) DO NOTHING`,
  [event.id, "stripe"],
);
if (rowCount === 0) return new Response(null, { status: 200 }); // already handled
await doTheWork(event);

Two details people get wrong.

Return 200 on the duplicate, not 409. A duplicate that you correctly ignored is a success from the sender's point of view. Returning an error makes the sender retry the thing you just told it you already have, and you have built a loop.

The ID must be the provider's, not yours. event.id from Stripe, X-GitHub-Delivery from GitHub, X-Shopify-Webhook-Id from Shopify. Hashing the payload body is a tempting substitute and a trap: two genuinely distinct events can have byte-identical bodies. Two charge.succeeded events for the same amount, same customer, one second apart, are two real charges. Dedupe by body hash and you just refused to fulfil an order somebody paid for.

Where a provider gives you no ID at all, you're inventing a synthetic key from whatever business fields are actually unique. That's a worse place to be, and it's worth a note in the code explaining the choice.

Pattern 2: Put the marker in the same transaction as the work

Pattern 1 has a hole. If the insert commits and then doTheWork throws, you have marked the event processed without processing it. The retry gets ignored and the event is silently lost, which is a strictly worse failure than the duplicate you were trying to prevent.

When the work is database work, close the hole with a transaction:

await db.transaction(async (tx) => {
  const { rowCount } = await tx.query(
    `INSERT INTO processed_events (event_id, provider)
     VALUES ($1, $2) ON CONFLICT DO NOTHING`,
    [event.id, "stripe"],
  );
  if (rowCount === 0) return;          // duplicate, commit an empty tx
  await tx.query(`UPDATE orders SET status = 'paid' WHERE id = $1`, [orderId]);
});

Either both land or neither does. This is the correct shape and you should reach for it by default.

Pattern 3: Claim-then-commit for side effects you can't roll back

Transactions don't extend to Resend, Salesforce, or a shipping API. Once the email is sent it is sent. Here the honest move is a two-phase marker:

CREATE TABLE processed_events (
  event_id     TEXT PRIMARY KEY,
  status       TEXT NOT NULL,          -- 'processing' | 'done'
  claimed_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  completed_at TIMESTAMPTZ
);

Claim the event before doing anything. Do the work. Mark it done. A concurrent duplicate arriving mid-flight sees processing and backs off rather than racing you.

The part that requires a decision: what happens to rows stuck in processing because the process died between claim and commit? You cannot know whether the email went out. Both choices are defensible and you should make it deliberately:

// ponytail: rows stuck in 'processing' >15min are re-claimed. Biases toward
// a possible duplicate email over a definitely-missing one. Flip this for
// side effects where duplication costs money (charges, shipments).
const STALE_CLAIM_MS = 15 * 60 * 1000;

For receipts and notifications, re-claiming is right, because a second email is cheap. For anything that moves money or goods, leave it stuck and alert a human. Encoding which one you chose, and why, in a comment next to the constant will save the next person a very confusing hour.

Pattern 4: Version guards for out-of-order delivery

Deduplication does not help you here, and this is the failure that actually corrupts data.

Retries mean event #2 can overtake event #1, and the backoff schedule is exactly what opens the gap: a delivery that waits five minutes lands after everything sent during those five minutes. A subscription is updated to active at 14:00:01 and to cancelled at 14:00:03. The first delivery fails and retries five minutes later. Your database now says active, and the customer keeps access to something they cancelled. Every event was processed exactly once. The result is still wrong.

Idempotency does not help here. What you need is a monotonic guard in the WHERE clause:

UPDATE subscriptions
SET status = $1, updated_at = $2
WHERE id = $3
  AND updated_at < $2;    -- refuse to apply stale state

Use whatever monotonic field the provider gives you: created on the Stripe event, a version field, an updated_at from the object itself. Not your own NOW(), which reflects when you got it, and that's precisely the ordering that's wrong.

For state machines with real invariants, go further and reject illegal transitions outright rather than relying on timestamps. An expired-to-active transition should raise, not silently apply.

A note on testing this

You cannot find these bugs by sending one webhook and checking the result. The tests worth writing are:

• Send the same event twice in a row. Assert one side effect
• Send the same event twice concurrently. This is the one that finds the missing unique index, and a sequential test will never fail
• Send two events out of order. Assert the newer state wins
• Kill the process between the marker write and the side effect. Assert what you decided in Pattern 3

The concurrent test is the valuable one. Most idempotency code is written assuming serial delivery and falls over the first time two retries land on two instances at the same millisecond. We wrote more about the gap between webhook tests that pass and webhook code that works in Your Webhook Signature Tests Prove Nothing.

Where AnyHook fits

AnyHook does not remove the need for idempotency on your side, and any service that claims it does is overselling. Duplicates originate at the sender and can arise on the network between us and you, which is not a layer anybody can eliminate.

What it does change is your ability to see the problem:

• Every delivery attempt is logged with the full request and response, so "did this fire twice, or did my handler run twice on one delivery" is answerable in the dashboard instead of by correlating three log sources at 2am
• Attempt numbers are explicit on every delivery, so a retry is distinguishable from a fresh event
• The AnyHook-Signature header carries a stable event ID across all retries of the same event, giving you a dedupe key even when the upstream provider is stingy with one
• Replay is deliberate and marked. Replayed events are flagged is_replay, so a backfill doesn't look like a mystery duplicate spike a week later
• The log is tamper-evident. When the question is "what exactly arrived, and was this row edited," a mutable database is a claim rather than evidence: A Tamper-Evident Webhook Log

Takeaway

Duplicates are not a defect in your provider, they're the price of never losing an event, and it's the right trade. Put a unique index on the provider's event ID today. Move the marker into the transaction when you touch the code next. And if you handle anything with a lifecycle (subscriptions, orders, deployments), add the version guard before out-of-order delivery writes a stale state you don't notice for a month.