A payment API times out and the client retries. How do you guarantee the customer is not charged twice?

What the timeout actually tells you

Nothing. A timeout means the client stopped waiting; it says nothing about whether the server processed the request. The charge may have been declined, may never have arrived, or may have completed and settled while the response was lost on the way back. Every design below exists because that ambiguity cannot be removed — you can only make it safe to resolve by asking again.

So the retry is not the problem to be prevented. The retry is correct behaviour, and the server's job is to make a second identical request produce the same single charge and the same answer as the first.

The key names the intent, not the attempt

The client generates an idempotency key when it forms the intention to charge, and reuses that exact key on every retry of that intention. Checkout attempt for order 8841 gets one key; a genuinely new purchase of the same basket tomorrow gets a different one. This is where implementations most often fail in a way that testing never catches, because a key minted inside the retry loop — a fresh UUID per HTTP attempt — makes every retry look like new business and the server dutifully charges again. The server cannot detect this; from its side the requests are unrelated.

Scope the key so that one tenant cannot collide with or probe another's: the uniqueness applies per key plus endpoint plus authenticated account, not globally. Store a hash of the request body alongside it, because the same key arriving with a different amount is not a retry, it is a client bug or an attack, and the correct answer is a conflict error rather than either charging the new amount or silently replaying the old one.

Insert first, do not look first

The natural-looking implementation is to select on the key, and if nothing is found, do the work and insert. That is a race, and under retries it is the likely path rather than an unlucky one: a client whose request timed out retries immediately, so two requests with the same key are in flight at overlapping times, both select, both find nothing, and both charge.

Let the database decide instead. Put a unique constraint on the scoped key and make the insert the claim on the work. Exactly one transaction can succeed; the loser gets a unique-violation error, which is not a failure but the signal that this request is a duplicate, and it branches into the retry-handling path. No application-level locking, no window between the check and the write.

CREATE TABLE idempotency_record (
  account_id     uuid    NOT NULL,
  endpoint       text    NOT NULL,
  key            text    NOT NULL,
  request_hash   bytea   NOT NULL,
  state          text    NOT NULL,   -- in_flight | completed
  response_status int,
  response_body  jsonb,
  payment_id     uuid,
  created_at     timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, endpoint, key)
);

The dedupe record and the effect commit together

A dedupe row that is written before the charge, or after it, is only an improvement in the odds. If the row commits and the process then dies, the payment never happens but every retry is deduped away, and the customer's order silently never gets paid. If the charge commits first and the row does not, retries charge again. The dedupe record only guarantees anything when it is committed in the same transaction as the effect it is deduping.

That is achievable when the effect is in your own database — the payment row, the ledger entries and the idempotency record are one COMMIT, and the response to be replayed is stored in the same write. It is not achievable when the effect is a call to an external provider, because you cannot enlist someone else's REST API in your transaction. The pattern there is two commits with a durable intent between them: insert the idempotency record in state in_flight together with a payment row in state pending and commit, then call the provider passing your own key through as its idempotency key, then commit the outcome and the stored response. The first commit means that whatever happens next, there is a record saying a charge with this key was attempted, which is what makes recovery possible. The provider's key handling is what makes the recovery call safe to make.

Answering a retry of a request still running

When the insert fails with a unique violation, read the existing row. If it is completed, replay the stored status and body verbatim — the client gets the original result, including the original payment identifier, and no second charge exists.

If it is in_flight, you genuinely do not know the outcome yet, and the two tempting answers are both wrong. Returning success is a lie you may have to retract, and it will be retracted after the client has already shown the customer a confirmation. Processing the request anyway is the double charge you built all this to prevent. The honest answer is to reject the retry as a concurrency conflict — HTTP 409 is the conventional choice, with a Retry-After hint — and let the client ask again shortly, at which point the row is completed and the real answer is available. A client-visible "we are still working on this, ask me again" is a much better contract than a guess.

Records need a retention window, and the window is a correctness parameter rather than a cleanup detail. Once a key's row is deleted, a retry carrying it is indistinguishable from new business, so retention must comfortably exceed the longest retry horizon any client, queue redelivery or operator-triggered replay could have. Consumer clients retry for seconds; an asynchronous job with backoff and a dead-letter queue can come back hours later.

Reconciliation is not a safety net you can skip

The request path can only enforce invariants about requests it saw. Several duplicate-charge routes are invisible to it. A process killed between the provider call and your second commit leaves a charge at the provider that your database has no completed record of. A provider outage that returns an error after taking the payment does the same. A customer who submits from two tabs generates two intents with two different keys, so key-based dedupe is working exactly as designed and there are still two charges. And a retry arriving after your retention window has expired will be treated as new.

Reconciliation closes those. On a schedule, pull the provider's transaction list or settlement file for the period and match it against your own records on the reference you supplied. Three buckets fall out, and each is a different problem: charges at the provider with no completed local record, which need to either be adopted into your ledger or refunded; local records marked complete with no provider charge, which is usually a bug in how you interpreted a response; and amount mismatches. Then run a business-level check for near-duplicate successful charges — same account, same amount, same merchant reference, close together — because that is the only place the two-tabs case is detectable, and it is a rule about customer behaviour rather than about HTTP.

The framing worth offering out loud is that idempotency keys make the known retry safe, and reconciliation is how you find out about the charges you never knew you made. A candidate who describes only the first half has designed a system that is correct whenever nothing crashes.

Idempotency is not the key header; it is the unique constraint and the shared commit underneath it. Everything else — replayed responses, 409 on an in-flight duplicate, the reconciliation job — follows from deciding that the dedupe record and the money must become durable at the same instant.