This content originally appeared on Level Up Coding - Medium and was authored by Vivek Mittal
How idempotency keys, dedup stores, and natural design let you retry safely — without charging a customer twice
Your payment request timed out. The spinner is still going, and you genuinely don’t know whether the money left the account or not. Now you — or worse, an automatic retry buried in some HTTP client — have to decide whether to press the button again. Press it, and you might pay twice. Don’t, and the order might never go through. Idempotency is the property that makes that decision boring: press it as many times as you want, and the result is the same as pressing it once.
This post covers what idempotency actually means, why distributed systems force you to care, the difference between operations that are naturally safe to repeat and ones you have to make safe, and how to build that safety with idempotency keys. It does not cover consensus or distributed locking — heavier tools for a different problem, and they get their own posts.
What is Idempotency?
Say your order service exposes a POST /charge endpoint. A customer taps "Pay," the server charges the card - and then their train enters a tunnel and the response never makes it back. The phone's HTTP library, doing exactly what it was told, retries. Now your server has two identical charge requests and no way to tell the honest retry from a genuine second purchase. Charge both and the customer pays twice. That's the whole problem in one sentence: a retry and a duplicate look identical to the receiver.
Idempotency solves this by making the operation’s effect independent of how many times it runs. The first POST /charge does the real work - it talks to the card gateway and moves the money. Every identical request after that produces the same outcome without redoing it: same response, same single charge, no new side effect. The server doesn't refuse the duplicate; it recognizes "I've already done this" and replays the answer it gave the first time.
Think of the call button for an elevator. You press it once and the car is on its way. The impatient person next to you jabs it nine more times, but the elevator arrives at the same moment regardless — the button’s effect is “come to this floor,” not “come nine times.” A button that dispatched a separate car per press would be a non-idempotent elevator, and you’d watch eight empty cars show up behind the one you needed. Idempotency is designing the button so the extra jabs cost nothing.
“An operation is idempotent if performing it multiple times produces the same result and the same observable side effects as performing it exactly once. The number of repetitions, beyond the first, changes nothing.”
The key insight is that idempotency doesn’t try to prevent duplicate requests — in a distributed system you can’t, because a duplicate is often indistinguishable from a legitimate retry. It makes duplicates harmless instead. The dangerous scenario — customer charged twice — becomes impossible not because the second request is blocked, but because it never re-runs the side effect; it only replays a result that already exists. You stop fighting duplicates and start absorbing them.
The Retry That Forces Your Hand
Duplicates aren’t a bug you can stamp out — they’re a direct consequence of how reliable systems are built.
Most messaging gives you at-least-once delivery: a message is guaranteed to arrive, possibly more than once, but never zero times. The alternative, at-most-once, drops messages on failure — unacceptable for a payment. So systems choose at-least-once and accept duplicates as the price of never losing data. Every queue I’ve worked with — Azure Service Bus, SQS, RabbitMQ — defaults to this; I covered the acknowledgment mechanics behind these redeliveries in the Message Queues post.
The concrete failure: a consumer (a service that reads messages off the queue) takes one, does the work, then crashes before it can tell the broker (the queue that holds and hands out messages) it finished. The broker never heard “done,” so it redelivers the same message to another consumer, which runs the work again. The same thing happens over HTTP, traced in the diagram below.
This is the trade-off that makes idempotency unavoidable: the moment you choose not to lose messages, you’ve signed up to handle duplicates.
Examples
- Stripe — every POST accepts an Idempotency-Key header, a unique string the client picks to tag one logical operation. Stripe saves the status code and body of the first request under that key and returns the same stored response for any retry, so a flaky connection can't create two charges. Keys are pruned after 24 hours.
- Square and PayPal — both expose a per-request key (idempotency_key for Square, a PayPal-Request-Id header for PayPal) so a retried payment or capture replays the original transaction instead of creating a second one.
- Kafka’s idempotent producer (enable.idempotence=true, the default since 3.0) - tags each record with a producer ID and a per-partition sequence number, so the broker silently drops a retried write it has already appended. It expects compatible settings (acks=all and a capped in-flight window) and quietly disables itself if you override them.
- Azure Service Bus duplicate detection — when enabled, the broker discards any message whose MessageId it has already seen inside a configurable time window, deduplicating at the infrastructure layer.
The thread through all of them: the system assumes retries will happen and is built to make them cost nothing. I used the same idea to keep the billing and notification consumers of a medical-imaging pipeline safe from duplicates — more on that below.
Problems Idempotency Solves
- Double charges and double writes — without it, a lost response or a redelivered queue message runs the payment twice, and the customer disputes a charge they only made once. With it, the retry replays the first result.
- Phantom side effects — sending the shipping label, the confirmation email, or the SMS twice. Each duplicate is a real-world action you can’t take back. Idempotency makes “send” mean “send once, ever, for this event.”
- Safe automatic retries — retry libraries like Polly only help if the operation underneath is safe to repeat. On a non-idempotent endpoint, an aggressive retry policy turns one transient blip into a pile of duplicate work.
- Crash recovery without reconciliation — a consumer that dies mid-processing can re-run the whole message on restart. If processing is idempotent, “just run it again” is a complete recovery strategy, no cleanup job required.
Natural vs Artificial Idempotency
Not every operation needs machinery to become idempotent — some already are. An operation has natural idempotency when repeating it lands on the same state no matter what. UPDATE account SET status = 'active' is naturally idempotent - run it once or fifty times, the status is active. So is DELETE FROM cart WHERE id = 42; after the first delete the row is gone, and every later delete is a harmless no-op. The pattern is assignment and removal: you're declaring a final state, not nudging the current one. This is why HTTP treats PUT and DELETE as idempotent while POST is not - PUT /user/42 {name: "Sam"} sets the resource to a known value, so repeats are safe, whereas POST /users means "create another one."
An operation needs artificial idempotency when its effect accumulates. UPDATE account SET balance = balance + 50 is the classic trap - every repeat adds another fifty dollars. "Append a row," "increment a counter," "send an email," "charge a card" all share this shape: doing it twice does twice as much. You can't rewrite the logic to fix this, because the business meaning genuinely is "do a thing." So you bolt idempotency on from the outside - attach a unique identifier to each logical operation and remember which ones you've already done.
The practical takeaway is to reach for natural idempotency first. If you can model a change as “set this to that” instead of “add this much,” repeats become free and you write no extra code. “The list should contain milk” is safe to repeat; “add milk to the list” buys you two cartons. Prefer the first kind of instruction whenever the design allows.
How Idempotency Keys Work
When an operation can’t be naturally idempotent, you give it an idempotency key — a unique identifier that names one logical operation and travels with every retry of that same operation; it’s the thread that ties a retry back to its original. The only hard rule: the same logical action reuses the same key, and two different actions never collide.
The key doesn’t have to be a GUID. A random UUID is the right default when nothing else identifies the operation — it’s what Stripe suggests, and its 128 bits make collisions negligible. But if your domain already has a stable identifier — an order ID, a payment intent, the study_id I key on below - use that. You can even derive the key from a hash of the request fields that define the operation, so the same logical request always produces the same key. The format doesn't matter; the property does: one stable value per operation, never reused for a different one.
The server keeps a dedup store — a database table with a unique constraint on the key, or a Redis entry with a TTL (time-to-live — it auto-expires). When a request arrives, the server writes the key into that store before doing any work, and the unique constraint means only one write per key can succeed. A successful write means a new key — mark it InProgress, do the work, then save the result as Completed. A failed write means the key is already there: if it's Completed, replay the saved result with no re-run; if it's still InProgress, another request is mid-flight, so the caller retries.
Why write the key before the work? Because if two copies arrive at once and you only check whether it’s done, both see nothing and both run it — a double charge. Writing the key first lets only one in; the other retries.
Implementing an Idempotent Consumer
Here’s the pattern in C#. The dedup store is fronted by a unique constraint on the key, which makes only one write per key succeed.
public async Task<PaymentResult> ChargeAsync(ChargeRequest req)
{
var key = req.IdempotencyKey; // #1
var record = await _store.TryBeginAsync(key); // #2
if (record is { Status: Status.Completed })
return record.Result; // #3
if (record is { Status: Status.InProgress })
throw new ConflictException("Retry shortly"); // #4
var result = await _gateway.Charge(req.Amount, req.Card); // #5
await _store.CompleteAsync(key, result); // #6
return result;
}
- #1 — The client supplies a stable key, the same one on every retry of this one logical charge.
- #2 — TryBeginAsync atomically inserts an in-progress row keyed on the idempotency key. Because the key column has a unique constraint, only one caller wins this insert; everyone else gets back the existing record.
- #3 — The key is already Completed, so replay the recorded result. No second call to the gateway, no second charge.
- #4 — The key exists but is still InProgress - a concurrent duplicate. Reject it with a conflict so the caller retries after the first finishes, rather than charging in parallel.
- #5 — Only a genuine first sighting reaches here. The side effect — the real charge — runs exactly once.
- #6 — Persist the outcome under the key so every future retry short-circuits at #3.
One honest caveat: the charge at #5 and the save at #6 are two operations. If the process crashes between them, the key is stuck in-progress while the money has already moved - the same dual-write hazard the Transactional Outbox post solves. Recovery means reconciling against the gateway or sharing a transaction where possible. Idempotency narrows the blast radius; it doesn't make atomicity free.
The Exactly-Once Illusion
“Exactly-once delivery” gets promised a lot, and believing it leads to real bugs. True exactly-once delivery — a message crossing the network exactly one time — isn’t achievable in general: the sender can never be sure its acknowledgment arrived, so it must either risk re-sending (at-least-once) or risk not sending (at-most-once). What systems like Kafka’s transactional producer actually give you is exactly-once processing: messages may still arrive twice, but the consumer’s effect on the world happens once.
The mechanism underneath that guarantee is idempotency. Kafka dedupes by sequence number, Service Bus by message ID, your service by idempotency key. “Exactly-once” isn’t a delivery miracle — it’s at-least-once delivery plus an idempotent receiver wearing a nicer label. Once that clicks, you stop hunting for a magic broker setting and start designing handlers that absorb repeats.
My Use Case — Deduplicating Study-Analysis Events
I had a system that analyzes medical imaging studies — X-rays, CTs, MRIs. When a study’s AI analysis finished, the worker published a StudyAnalysisCompleted event to a Service Bus topic, and four independent consumers reacted: a results API, notifications, billing, and audit. The topic gives at-least-once delivery, so any consumer crash or slow acknowledgment could redeliver the same event - and for billing, a duplicate meant charging for the same study twice.
Each consumer had to absorb repeats on its own. I keyed every handler on the study_id and put a small guard in front: it wrote the ID into a dedup table protected by a unique constraint, did the real work only on the first sighting, and on any redelivery acknowledged the message without re-running a thing. Billing was the one I couldn't get wrong - a duplicate row there was a real charge someone would later have to dispute and reverse. Natural idempotency wasn't an option (you can't model "send a notification" as a state assignment), so the key-based guard was the only safe path.
When to Reach for Idempotency
- Money or irreversible side effects — charges, refunds, transfers, sending physical mail. Anything you can’t cleanly undo needs to be safe against repeats.
- Anything behind a message queue — at-least-once delivery means redelivery is a when, not an if. Queue consumers should be idempotent by default.
- Public-facing POST/PATCH APIs - clients retry on timeout, and you don't control their retry logic. An idempotency key lets them retry without fear.
- Webhooks you both send and receive — providers redeliver on any non-2xx, so receivers must dedupe; senders should give consumers a stable event ID to dedupe on.
You don’t need this when the operation is already naturally idempotent — a pure read, a PUT that sets a value, a DELETE of a known resource - or when a duplicate is genuinely harmless. Bolting a dedup store onto something that's safe to repeat anyway just adds storage, latency, and a new failure mode for no benefit. Idempotency is insurance; don't buy a policy on something that can't get hurt.
Idempotency vs Related Ideas
- Idempotency vs deduplication — deduplication is one implementation of idempotency: remember IDs you’ve seen and drop repeats. Naturally idempotent operations get the same safety with no dedup store at all — repeating them simply doesn’t accumulate.
- Idempotency vs at-least-once delivery — partners, not rivals: at-least-once produces duplicates, idempotency neutralizes them.
- Idempotency vs distributed locking — a lock prevents two things from running at once; idempotency makes it safe if they do. Locks are pessimistic and expensive; an idempotent handler is usually the lighter, more resilient answer when all you need is to stop duplicate processing.
Issues with Idempotency
- The dedup store is unbounded without a TTL — keep keys forever and the table grows without limit; expire them too soon and a late retry sails through as a “new” request and double-acts. Stripe’s 24-hour pruning bets that legitimate retries arrive well inside a day. You’re picking a window, and it has teeth on both sides.
- The in-flight race is easy to get wrong — as the code showed, checking only for a completed key leaves a gap where two concurrent duplicates both execute. Closing it means writing the key as in-progress before the work, plus a sensible answer for the caller who loses the race.
- Every request now pays an idempotency tax — the dedup store sits on the hot path of every write, not just the duplicates. Each request does an extra indexed insert or lookup before any real work begins, and at high throughput that store becomes its own contention point — the safety mechanism turns into the bottleneck.
- Key reuse is silent and dangerous — if a client accidentally reuses a key for a genuinely different operation, the server returns the old result and the new operation never runs. The customer thinks they bought a second item; they got a replay of the first. Stripe guards against this by comparing request parameters and erroring on a mismatch — worth copying.
- Stale replayed responses — you’re returning a result captured at first execution. If the underlying resource changed since, the replay can be out of date. For a payment confirmation that’s fine; for anything time-sensitive, the staleness is a real correctness question.
- It doesn’t grant atomicity — idempotency makes a repeated operation safe, not a multi-step one all-or-nothing. The crash-between-charge-and-save window is still yours to handle, usually with the outbox pattern or a reconciliation pass.
- Key generation is now a client responsibility — the guarantee is only as good as the client’s discipline in producing one stable key per operation and reusing it across retries. A client that generates a fresh key on every attempt protects nothing.
Wrapping Up
Idempotency feels like a footnote until the night a redelivered message charges a customer twice and you’re explaining the refund. It matters for a structural reason: the same engineering that keeps you from losing data is what hands you duplicates, so every reliable system eventually has to decide how it absorbs repeats. The ideas to carry away:
- An idempotent operation runs any number of times and leaves the same result and side effects as running once — it makes duplicates harmless instead of trying to prevent them.
- Duplicates are unavoidable: at-least-once delivery and lost responses guarantee retries, and a retry is indistinguishable from a genuine duplicate at the receiver.
- Model a change as state assignment (PUT, DELETE, "set status to active") and idempotency comes free; accumulating operations (charge, increment, send) need a key plus a dedup store - written before the work to close the concurrent-duplicate window.
- “Exactly-once” is just at-least-once delivery plus an idempotent receiver; there is no exactly-once delivery at the network layer.
- The safety isn’t free: a TTL window to size, an in-flight race to handle, silent key reuse to guard against, and atomicity it narrows but never removes.
The trade-offs are real, but the insight is worth holding onto: in a distributed system you can’t stop the same thing from happening twice, so stop trying. Design so the second time costs nothing. The elevator button that ignores the extra jabs isn’t broken — it’s the only one that works at scale.
Written & edited by — Vivek Mittal
https://www.linkedin.com/in/vivekmittal06/
Additional Reading
- Message Queues — What Happens Between Send and Receive
- Transactional Outbox — The Cost of Guaranteed Event Delivery
- Stripe API — Idempotent Requests
Idempotency — Why Running It Twice Should Change Nothing was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Vivek Mittal
Vivek Mittal | Sciencx (2026-06-24T14:57:42+00:00) Idempotency — Why Running It Twice Should Change Nothing. Retrieved from https://www.scien.cx/2026/06/24/idempotency-why-running-it-twice-should-change-nothing/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.