Making a Payment Retry Safe

In payments, a retry is the normal case. A client times out and sends the request again. A provider delivers the same callback twice. Two workers pick up the same job. If any of these moves money a second time, a real person loses real money. I use two mechanisms together: an idempotency key at the edge of the system, and a versioned write at the ledger. Each one covers a failure that the other misses.

What goes wrong without them

Here is the most common double charge. It needs only a slow network:

1. The client sends "charge 1,000" to my API.
2. My API calls the card provider. The provider charges the card.
3. The response is slow, and the client times out.
4. The client retries. My API sees a new request and charges again.

The second failure is quieter. Callbacks for two different payments to the same wallet arrive at the same time. Both read the balance as 0, both add 1,000, and both write 1,000. The wallet now says 1,000 when it should say 2,000. Nothing crashes, and nothing logs an error.

The first failure is about the same request arriving twice. The second is about two writers racing on the same row. They need different fixes.

The shape of a safe request

1. 1. Claim the key
   

   Insert the key with my attempt ID. A duplicate gets the stored result.

2. 2. Call the provider
   

   Pass the same key, so the provider also deduplicates.

3. 3. Fenced result
   

   Mark the key done only if my attempt still owns it.

4. 4. Versioned write
   

   Same transaction: update the balance only if the version is unchanged.

Mechanism 1: the idempotency key

The client makes up a unique key for each logical operation and sends it with every attempt of that operation. The server stores the key before it does any work. The database, not the application code, decides who owns the key:

CREATE TABLE idempotency_keys (
  key           text PRIMARY KEY,
  request_hash  text NOT NULL,
  attempt_id    uuid NOT NULL,
  status        text NOT NULL CHECK (status IN ('in_progress', 'done')),
  response      jsonb,
  created_at    timestamptz NOT NULL DEFAULT now()
);

type Claim =
  | { kind: "new" }
  | { kind: "replay"; response: unknown }
  | { kind: "busy" };
 
async function claimKey(
  db: Db, key: string, requestHash: string, attemptId: string,
): Promise<Claim> {
  const inserted = await db.query(
    `INSERT INTO idempotency_keys (key, request_hash, attempt_id, status)
     VALUES ($1, $2, $3, 'in_progress')
     ON CONFLICT (key) DO NOTHING
     RETURNING key`,
    [key, requestHash, attemptId],
  );
  if (inserted.rowCount === 1) return { kind: "new" };
 
  const { rows } = await db.query(
    `SELECT request_hash, status, response FROM idempotency_keys WHERE key = $1`,
    [key],
  );
  const row = rows[0];
  if (row.request_hash !== requestHash) {
    throw new ConflictError("Same key, different request body");
  }
  return row.status === "done"
    ? { kind: "replay", response: row.response }
    : { kind: "busy" };
}

Three details carry the weight:

• ON CONFLICT DO NOTHING, not "select, then insert". Two attempts that arrive in the same millisecond cannot both win a primary key.
• The request hash. A client that reuses a key with a different amount gets an error, not the old response.
• The busy state. A retry that arrives while the first attempt still runs must not start a second one. It gets a "try again later" answer. A key that stays in_progress past a timeout means the first attempt died or stalled. Recovery takes the key over with a conditional update that swaps in its own attempt ID, then asks the provider about that key. It never charges again.

The same key goes to the provider when the provider supports one. That closes the gap in step 2 above: if my process dies after the provider charged the card, the next attempt asks the provider again with the same key and gets the first charge back.

Provider callbacks get the same treatment with the provider's event ID as the key. A unique constraint on that column is enough.

Mechanism 2: the versioned ledger write

An idempotency key stops one request from running twice. It does nothing when two different requests, or a request and a callback, change the same balance at the same time. For that I put a version number on every row that holds money state, and every write says which version it read:

async function applyCredit(db: Db, walletId: string, amount: bigint): Promise<void> {
  for (let attempt = 0; attempt < 3; attempt++) {
    const { rows } = await db.query(
      `SELECT balance, version FROM wallets WHERE id = $1`,
      [walletId],
    );
    const balance = BigInt(rows[0].balance); // pg returns bigint as a string
    const version: number = rows[0].version;
 
    const updated = await db.query(
      `UPDATE wallets
          SET balance = $1, version = version + 1
        WHERE id = $2 AND version = $3`,
      [balance + amount, walletId, version],
    );
    if (updated.rowCount === 1) return;
    // Another writer changed the row after we read it. Read again.
  }
  throw new ConcurrencyError(`wallet ${walletId}: too many concurrent writes`);
}

If another writer got there first, the WHERE version = $3 matches no row, rowCount is 0, and nothing is overwritten. The losing writer reads the new state and tries again, or gives up loudly. A lost update becomes a visible conflict.

In a real ledger, the balance update and the ledger entry go into one transaction, and the entry is append-only. The balance is a cached sum of the entries.

The result on the idempotency key commits in that same transaction, and the key row is the fence:

await db.transaction(async (tx) => {
  const owned = await tx.query(
    `UPDATE idempotency_keys SET status = 'done', response = $3
      WHERE key = $1 AND attempt_id = $2 AND status = 'in_progress'`,
    [key, attemptId, response],
  );
  if (owned.rowCount === 0) throw new LostOwnershipError(key); // rolls back
  await applyCredit(tx, walletId, amount); // balance update; the ledger entry is left out
});

A crash before the commit leaves neither the credit nor the result, so recovery can run the ledger step again. A crash after it leaves both, and a retry gets the stored result. There is no moment where the ledger moved and the key still says in_progress. A stalled first attempt that wakes up after recovery took over no longer owns the key. Its update matches no row, its transaction rolls back, and the ledger keeps one credit.

The rule that ties it together

Both mechanisms serve one business rule: a retry never moves money twice, and a concurrent write never loses an update. I write that rule down before any code, and I test it directly. The tests:

• The same request sent twice, in parallel, gives one charge.
• The same callback delivered twice gives one ledger entry.
• A crash between the provider call and the result write, followed by a retry, gives one charge and one ledger entry.
• A stalled first attempt that finishes after recovery took over gives one ledger entry.
• Two callbacks for one wallet at the same time give the correct sum.

Alternatives I rejected

• Check in application code, then insert. "Does a payment with this key exist? No? Then create it." Two requests can both pass the check before either inserts. Only a database constraint closes that window.
• Lock the row for the whole request. SELECT ... FOR UPDATE works, but the request also calls a provider over the network. Holding a row lock across a call that can take seconds turns one slow provider into a queue of blocked requests.
• A distributed lock in a cache. A lock with a timeout can expire while its owner is paused, and then two owners act at once. Unless every write also checks a fencing token, the lock only makes the race rarer. The version column and the attempt ID on the key are those fencing tokens, so I use them directly.
• Trust the provider's idempotency alone. It protects the call to the provider. It does not protect my own ledger from a callback that arrives twice, or from two of my workers.

What I would do differently

I would put the idempotency key into the API contract on day one, as a required header. Without it, the server has to guess which requests are retries, and a wrong guess is a double charge.

I would also run the reconciliation between the ledger and the provider reports from the first week, not after the first incident. The two mechanisms above prevent the failures I can think of. The reconciliation catches the ones I did not think of.