Webhooks Are Hostile: Making Payment Events Idempotent, Verified, and Ordered

The naive version of handling a payment webhook is one line of intent: "when the deposit-confirmed event arrives, credit the user." That single line is a money bug waiting to happen, because it assumes the event arrives exactly once, in order, from who it claims to be, at a moment when your system is ready for it. None of those assumptions hold. Webhooks arrive more than once. They arrive out of order. They arrive forged. They arrive while your database is mid-write on the same account. Build on "it will arrive once, correctly" and you will double-credit a balance on a provider retry, or credit a balance because someone POSTed a fake event to your public endpoint.

Picture the simplest failure. A deposit confirms, the provider sends the webhook, you credit the account, and your 200 response is lost to a network blip. The provider, never having heard a clean acknowledgement, retries. Your naive handler credits the account again. The user now has double the money, and you find out at the end of the month when the books do not balance. Every rule below exists to make that story impossible.

Verify first

Before you parse anything, verify the provider's signature, almost always an HMAC of the request body with a shared secret. The trap that catches everyone: you must hash the raw bytes exactly as received, not a re-serialized version of the parsed JSON. Many frameworks helpfully parse the body for you, and re-stringifying it reorders keys or changes spacing, so your signature check fails on legitimate events while you stare at it for an hour. Capture the raw body, verify, then parse. Reject anything that fails without ceremony, because an unverified webhook is just an anonymous internet request asking you to move money.

Dedupe on the event id

Every reputable provider gives each event a stable id. Record the ones you have processed, and if you see a repeat, acknowledge it and do nothing. Crucially, do not rely on an in-memory set or a check-then-write, because two retries can race through the gap between the check and the write. Put a uniqueness constraint on the event id in the database so the duplicate insert simply fails, and make the credit itself idempotent, an upsert or a ledger entry keyed on the event, so even a race that slips past the first guard cannot apply the money twice. Idempotency is belt and braces here on purpose.

Model state as a transition, not a flag

A payment is not a boolean, it is a state machine, and webhooks for the same payment arrive out of order. Model the legal transitions explicitly and reject the illegal ones, so a late "confirmed" cannot resurrect something you already marked failed, and a duplicate "settled" is a no-op rather than a second payout. The state machine is also what makes out-of-order delivery a non-event instead of a crisis: you are not applying webhooks, you are applying transitions against current state.

Ack fast, process async

Persist the raw event, return 2xx immediately, and do the real work, crediting, notifying, downstream calls, on a queue with retries and a dead-letter. This is not just tidiness. Providers enforce a timeout on your endpoint, and if you do slow work inline and exceed it, they assume failure and resend, turning one event into a duplicate storm precisely when your system is already slow. Fast acknowledgement is back-pressure protection, and the queue is what lets you absorb a burst of events without dropping any.

Designing the dedupe store

The dedupe check sounds trivial until you build it under real concurrency, where two copies of the same event arrive within milliseconds of each other and a naive "have I seen this id?" lookup lets both through the gap before either writes. The reliable design pushes the guarantee down to the database: a unique constraint on the provider's event id means the second insert fails atomically, no matter how the two requests race, and you treat that failure as "already processed" rather than an error. You decide how long to retain processed ids based on how far back a provider might realistically retry, because keeping them forever is wasteful and expiring them too soon reopens the duplicate window. And you make the side effect, the credit, idempotent in its own right through the ledger, so that dedupe and idempotent application are two independent guards rather than a single point of failure. Belt and braces is not paranoia here, it is the minimum for money.

Reconciliation is not optional

Keep every raw event you receive. You will want it for disputes, for debugging signature failures, and for replaying after a bug. And accept the uncomfortable truth that webhooks are best-effort: providers drop them, networks eat them, and your endpoint has downtime during exactly the deploy when an event fires. So you also poll the provider's API on a schedule to reconcile, comparing what they think happened against what your ledger recorded. The provider's ledger is the source of truth; your webhook handler is a fast-path cache of their reality, not the system of record. When the two disagree, the provider wins and you correct, and you treat a recurring disagreement as a bug to find, not a row to quietly patch.

Replay and recovery

Storing every raw event is not just good hygiene, it is what makes recovery possible when, not if, you ship a bug in the processing path. Because the raw events are durable and processing is idempotent, fixing a mistake becomes a replay: you correct the consumer and re-run the affected events, and idempotency guarantees the ones that already applied correctly are no-ops while only the genuinely broken ones get fixed. That property turns a class of incidents that would otherwise mean manual database surgery into a routine, auditable operation. It also means a new feature can be backfilled by replaying history through it, and a dispute can be investigated by replaying exactly what the system saw. The raw event log is cheap to keep and repeatedly turns out to be the thing that saves you.

Watch for the warnings

All of this is invisible until it is on fire, so you make it visible. Alert when the dead-letter queue grows, because that means events are failing to process and money is in limbo. Alert when reconciliation finds drift, because that is your earliest signal that a provider changed a payload or a code path is mishandling a state. The teams that get burned by webhooks are not the ones who lacked a rule, they are the ones who had no idea anything was wrong until a customer or an auditor told them.