When the same automation runs twice

Click once, wait, worry, click again. That small human ritual is enough to expose a large systems problem. Reliable automation needs to recognize the same piece of business even when it arrives wearing a fresh request ID.

Retries keep business processes moving. Without a reliable way to recognize the same operation, they can also create duplicate orders, invoices, notifications, or payments.

A customer submits an order. The browser waits, then reports a timeout. The customer tries again. Somewhere behind the interface, a background worker retries the job after losing its connection.

Did the first attempt fail, or did it finish without sending a response?

That uncertainty is normal in distributed systems. The expensive mistake is assuming that an unanswered request means the business operation never happened.

One business decision, several deliveries

Consider a hypothetical wholesale portal. Its order-submission workflow records an order, schedules a confirmation, and asks a payment provider to authorize a transaction. The message carrying that work might arrive twice after a worker restarts. Two application instances might also receive the same retried request at nearly the same moment.

Amazon SQS standard queues explicitly support at-least-once delivery, meaning a consumer may receive the same message more than once. AWS also recommends making Lambda functions idempotent because duplicate event processing can occur. These are expected properties to design around, not proof that a queue or runtime has failed. [1] [2]

What matters is whether the business produces one order and one intended set of side effects.

Give the operation its own identity

An HTTP request ID describes one attempt. A business operation ID describes the work those attempts are trying to accomplish. The distinction is crucial.

Assign a stable, suitably unique operation identifier when the order is first submitted. A retry carries that same identifier. A genuinely new order receives a new one. Scope the identifier appropriately—for example, to a customer or tenant—and associate it with the original request's essential parameters.

If a retry repeats the same operation, return its existing status or outcome. If the same identifier arrives with conflicting parameters, reject it rather than silently attaching different work to an earlier result.

An idempotency key alone is not the protection. The application needs durable state behind it.

Put the guard where concurrent workers cannot outrun it

A simple “check whether the order exists, then insert it” can fail when two workers check simultaneously. Both see nothing; both proceed.

For a database-backed workflow, enforce a unique business-operation key in the database itself. In a transactional engine, coordinate the key registration, order creation, and durable work state so competing attempts cannot each claim to be first. Return a clear in progress status if the first attempt is still running; do not create a second order just because its response is pending.

MySQL supports unique constraints, which provide a stronger concurrency boundary than an application-only existence check. The exact transaction, locking, and retry design still depends on the workflow. [4]

External services require another boundary

A database transaction cannot roll back an email already sent or a payment action already accepted by another company.

One useful design is a transactional outbox: commit the business record and an instruction to perform its external side effect in the same local transaction. A separate worker sends that instruction and records progress. The outbox itself can be retried, so the receiver or sender must also handle duplicates.

Where an external API supports idempotency keys, reuse the appropriate key when retrying the same operation. Stripe documents this pattern for payment-related API calls, with defined retention and behavior that vary by API version. Provider-specific guarantees should be checked rather than assumed to last forever. [3]

If an external call times out after being accepted, first recover or reconcile its status when possible. Blindly generating a fresh operation ID can turn uncertainty into a duplicate charge.

Test the ugly middle, not only the happy path

A repeatable acceptance test should use synthetic orders and a sandbox provider. Submit an operation once, replay it, and confirm there is one business result. Send two matching attempts concurrently. Simulate a timeout immediately after the database commit. Restart the worker before the outbound action is acknowledged. Try the same key with altered parameters and confirm that it is rejected.

Then inspect the evidence: operation ID, attempt ID, outcome, durable record count, queued work, external action identifier, and final status. Record retries and suppressed duplicates without logging secrets or sensitive payloads.

The goal is measurable: one intended business operation produces one authorized business outcome, even when the transport delivers work more than once. Unresolved or repeatedly failing operations should surface to an owner instead of disappearing into an endless retry loop.

A short review checklist

Before trusting an automated business workflow, ask whether it has a stable operation identifier, a database-enforced uniqueness boundary, a defined response for concurrent or conflicting retries, a safe pattern for external side effects, and a sandbox test proving the result after partial failure.

Retries are a resilience mechanism. Their safety comes from the business rules around them.