Idempotency and Safe Retries

Part seven of the series, and if you remember one pattern from this whole book, make it this one. The async chapter ended on a warning: retries mean duplicates. This chapter is the answer. Idempotency — the property that doing something twice has the same effect as doing it once — is the single most useful idea for surviving an unreliable network, and the network is always unreliable.

Why duplicates are inevitable

Here’s the uncomfortable truth underneath all distributed communication: you can never know for certain that your request arrived. A client sends “charge ₱500” and the response never comes back. What happened?

Client ──"charge ₱500"──▶ Server
Client ◀─────  ??????  ─── Server   ← timeout. Which of these happened?

  (a) request lost, never charged           → must retry
  (b) charged, but the response was lost     → must NOT retry (double charge!)
  (c) charged, response coming, just slow    → must NOT retry

The client cannot distinguish these. The timeout looks identical in all three. So it faces an impossible choice: retry (and risk double-charging in case b) or give up (and risk never charging in case a). This isn’t a bug you can fix — it’s a fundamental property of messages over a network. Every layer that “guarantees delivery” does so by retrying, which means every such layer delivers at-least-once: it will, eventually, deliver twice.

The escape hatch isn’t making delivery exactly-once (impossible). It’s making the receiver not care how many times the message arrives. That’s idempotency.

What idempotent actually means

An operation is idempotent if applying it multiple times has the same effect as applying it once. Some operations are naturally idempotent; some are naturally not:

The dangerous ones are the “relative” and “create” operations — += 500, INSERT, POST /charge. These are exactly the operations that matter most (money, orders), and exactly the ones a retry corrupts. So we engineer idempotency onto them.

The idempotency key

The standard technique — the one Stripe, PayPal, and every serious payments API use — is the idempotency key. The client generates a unique key per logical operation and sends it with the request. The server records which keys it has already processed and, on a repeat, returns the original result instead of doing the work again.

public function charge(ChargeRequest $request)
{
    $key = $request->header('Idempotency-Key');   // client-generated, one per logical charge

    // First writer wins; a duplicate key collides on the unique index and we short-circuit.
    $record = IdempotencyKey::firstOrCreate(
        ['key' => $key],
        ['status' => 'processing'],
    );

    if (! $record->wasRecentlyCreated) {
        // Seen this key before → return the stored result, do NOT charge again
        return response()->json($record->response, $record->status_code);
    }

    $result = $this->paymentGateway->charge($request->amount);   // the real, un-repeatable work

    $record->update([
        'status'      => 'done',
        'response'    => $result->toArray(),
        'status_code' => 200,
    ]);

    return response()->json($result, 200);
}

Now the client can retry the timed-out charge as many times as it likes. The first request that lands does the work and stores the result under the key; every retry with the same key gets that stored result back. Case (b) from earlier — charged but response lost — is now safe: the retry returns the original charge’s response instead of charging again. The key’s uniqueness constraint in the database is what makes this a race-free “first one wins.”

Idempotency for queue jobs

The same problem hits async jobs — at-least-once delivery means a job can run twice. Same solution, applied at the job:

public function handle(): void
{
    // Guard on a natural key: has THIS order's confirmation already been sent?
    if ($this->order->confirmation_sent_at !== null) {
        return;   // a previous (possibly crashed-after-success) run already did it
    }

    Mail::to($this->order->email)->send(new OrderConfirmation($this->order));
    $this->order->update(['confirmation_sent_at' => now()]);
}

Often you don’t even need a separate keys table — a natural marker on the row (confirmation_sent_at, a status transition, a unique constraint on (order_id, type)) does the job. The principle is identical: before doing the un-repeatable work, check whether it’s already been done.

“Exactly-once” is a lie you engineer around

You’ll see systems advertise “exactly-once delivery.” It doesn’t exist at the messaging layer — physics won’t allow it. What does exist is at-least-once delivery + idempotent processing = effectively-once, which is the same thing from the outside. Nobody can tell your idempotent handler ran twice, because the second run was a no-op. That’s the trick: you don’t prevent the duplicate, you make it harmless. Stop looking for exactly-once delivery; build effectively-once processing.

Caveats and best practices

• Make every write endpoint and every job idempotent by default. Anywhere a retry could fire — behind a queue, a payment flow, a webhook receiver, a mobile client on flaky signal — a non-idempotent write is a latent double-action bug. This is not an optimization; it’s correctness.
• The client generates the key, not the server. The whole point is that a retry reuses the same key. If the server generated it, each retry would look new. Generate it once per logical action (e.g. a UUID when the user taps “Pay”), reuse it across retries.
• Prefer natural idempotency where you can get it. SET status = 'paid' is inherently idempotent; INSERT payment is not. A unique constraint on (order_id) turns a duplicate insert into a caught error instead of a second row — let the database enforce it.
• Store the original response, and mind the in-flight race. Two identical requests can arrive simultaneously; the unique key insert is what serializes them. Return the stored result on repeats so retries are not just safe but identical.
• Idempotency keys need a TTL. You don’t keep them forever — expire them after a reasonable window (24h is common for payments). Long enough to cover any realistic retry, short enough not to grow unbounded.
• This is the foundation for optimistic UI. Optimistic writes assume success and retry on failure — which is only safe if the retry is idempotent. The two patterns are partners.

Conclusion

Truth → the network can't do exactly-once; every reliable channel is at-least-once
So → the same request WILL sometimes arrive twice (timeout ≠ "it failed")
Fix → idempotency: doing it twice == doing it once
How → idempotency key (client-generated) + stored result; or a natural marker on the row
Result → at-least-once delivery + idempotent processing = effectively-once

The network will deliver your important request twice on its worst day, and you don’t get to prevent it — so the entire discipline is making the second delivery a no-op. Idempotency keys for endpoints, natural markers for jobs, “check before you do the un-repeatable thing” everywhere. Build it in by default and a whole category of double-charge, double-email, double-order incidents simply can’t happen. Next section — Speed & Failure — opens with caching patterns and their failure modes.