Idempotency keys sound like the sort of thing you only need once you are running at proper scale, with a team of platform engineers and a name for your incident process. They are not. I have added them to systems with a handful of paying customers because the alternative was a support inbox full of "why was I charged twice" emails. If your product touches payments, bookings, or anything else where doing the same thing twice actually costs someone money, this is not optional, it is just plumbing you have not built yet.
The idea is simple once it clicks. A request from the outside world, a customer clicking pay, a mobile app retrying a failed call, a webhook you are processing, gets tagged with a unique key. Your system remembers which keys it has already handled and what it did the last time. See the same key twice, and instead of doing the work again, you just hand back the original result. Same outcome, no matter how many times the request actually lands.
Why This Keeps Happening Even When Your Code Is Fine
The instinct is to blame duplicate requests on bad client code, and sometimes that is fair. But the more common cause is far less avoidable: networks are unreliable in exactly the way that makes this hard. A customer clicks pay, the request reaches your server, you process the payment, and then the response gets lost on the way back. As far as the client knows, the request failed, so it retries. As far as your system is concerned, everything worked the first time, and now it is about to work again.
You cannot code your way out of that by being more careful. The client genuinely does not know whether the first attempt succeeded, and retrying is the correct thing for it to do. The fix has to sit on your side, treating "did this exact request already happen" as a question your system can actually answer.
Where I See Small Teams Get Caught Out
Trusting the Database Transaction to Save Them
A transaction stops your own database ending up in an inconsistent state. It does nothing about a second, entirely separate request arriving thirty seconds later and running the exact same logic from scratch. I have seen teams treat "we wrapped it in a transaction" as the whole answer, and it is only ever half of it.
Building Retries Without Building Idempotency First
This is the one that actually causes the damage. Retry logic on its own is a good idea, the same discipline I wrote about when covering webhooks for small SaaS teams, but adding retries before the receiving end can safely handle the same request twice just multiplies the number of duplicates you generate. Retries and idempotency are a pair. Ship one without the other and you have made the problem worse, not better.
Using the Request Body Instead of a Real Key
Hashing the payload and calling that your idempotency key feels clever until a completely legitimate second charge for the same amount, to the same customer, on the same day, gets silently swallowed because it looks identical to the first one. The key needs to represent the client's intent to perform this specific action once, not the shape of the data inside it.
How I Actually Build This
Let the Client Generate the Key
The client, whether that is a browser, a mobile app, or another service calling your API, generates a unique key (a UUID is fine) the moment the user takes the action, and sends it as a header on every attempt, including retries. This matters because only the client actually knows whether an attempt is a genuinely new action or a retry of one that might already have gone through.
Store the Key With the Result, Not Just a Flag
Do not just record that a key has been seen. Store the key alongside the actual response you gave back, ideally in the same transaction as the work itself. When the same key turns up again, whatever the reason, you return that stored response rather than running the logic a second time. This is the detail that turns idempotency from a nice idea into something that actually works under load.
Give Keys a Sensible Lifetime
Keeping every idempotency key forever is wasteful and rarely necessary. Twenty four hours covers almost every retry scenario I have ever actually seen, from a flaky mobile connection to a customer refreshing a stuck page. Pick a window, document it, and let old keys expire.
Make the Failure Case Idempotent Too
It is tempting to only bother with all this on the success path. Failures need the same treatment. If a request fails validation, that is a stable outcome too, and replaying the same key should give you the same rejection, not a fresh attempt at the same broken request.
My Honest Take
Idempotency keys are one of those pieces of infrastructure that customers will never notice you built, right up until the one time you did not, and someone gets charged twice for a booking that failed. I would rather spend half a day on this early, in the same spirit as the approach I take to database migrations without downtime, than spend a Friday afternoon manually refunding customers and writing an apology email.
If you are building payment flows, booking systems, or any API where a retried request could cause real damage, this is exactly the kind of unglamorous but essential problem I help teams work through via software development consulting. It is the same theme running through The 28 Day Startup, the boring foundations are the ones that actually protect your customers, and your reputation, once real money starts moving through the system.
