A payment request leaves your system.
The provider receives it.
The provider processes the transaction successfully.
Then the network connection disappears before the response reaches you.
What happened?
Many systems are tempted to answer:
FAILED.
But that’s not what happened.
The truth is more uncomfortable.
You don’t know.
That distinction sounds small.
In a payment system, it can be the difference between safe recovery and charging someone twice.
A Timeout Is Not a Failure
Working with payment platforms taught me to be very careful with the word “failed.”
A timeout tells you something specific:
You did not receive a response within the expected period.
It does not necessarily tell you what happened on the other side of the network.
This matters because distributed systems don’t fail neatly.
Imagine this sequence:
Your gateway creates a payment request.
The request reaches the provider.
The provider successfully processes £500.
The provider sends the response.
The network fails.
Your gateway sees a timeout.
If your architecture translates that timeout directly into:
PAYMENT FAILED
you have made an assumption.
And if your next action is an immediate retry, that assumption can become expensive.
The Retry Trap
Retries are one of the most useful reliability mechanisms in distributed systems.
They’re also one of the easiest to misuse.
For many operations, retrying is harmless.
For financial operations, every retry deserves more thought.
If the first request actually succeeded, what will the second request do?
Will the provider recognise it as the same transaction?
Will it create another financial effect?
Can you query the original transaction before attempting another operation?
These aren’t implementation details.
They’re business questions expressed through architecture.
That is why I don’t think about retry as:
“Something failed. Try again.”
I think about it as:
“Something is uncertain. Establish what we know before deciding what should happen next.”
Give the Transaction an Identity
One of the foundations of safe recovery is transaction identity.
Before sending an important financial operation externally, establish a stable identity for the business transaction.
That identity should survive:
Network retries.
Browser refreshes.
Client reconnects.
Duplicate callbacks.
Repeated requests.
Suppose a customer presses Pay twice.
You may receive two HTTP requests.
But that doesn’t necessarily represent two business intentions.
The architecture needs a way to distinguish:
Another request
from
Another payment.
Those are not the same thing.
Idempotency Solves a Different Problem
This is where idempotency becomes important.
Retries and idempotency are related, but they solve different problems.
Retry asks:
Should I attempt this operation again?
Idempotency asks:
If I receive the same business operation again, can I prevent another unintended effect?
That distinction is important.
A well-designed idempotent payment operation allows repeated delivery of the same business intent without repeatedly creating the financial outcome.
Multiple requests.
One transaction.
One intended financial effect.
But idempotency alone does not remove every uncertainty.
You still need to understand transaction state.
Don’t Pretend You Know
This leads to a design decision I strongly favour.
Model uncertainty explicitly.
Instead of forcing every transaction immediately into:
SUCCESS
or
FAILED
allow states that reflect reality.
For example:
CREATED
PENDING
SUCCESS
FAILED
UNKNOWN
SETTLED
REVERSED
REFUNDED
The exact state machine will depend on the payment model, but the principle is more important than the names.
Don’t hide ambiguity.
If the system genuinely doesn’t know whether an external operation completed, UNKNOWN may be more correct than FAILED.
Now you can build recovery around the truth.
Resolve Before You Repeat
For a timed-out payment, a safer recovery path might look like this:
First, use the original transaction identity.
Then query the provider if status lookup is supported.
If the provider confirms success, continue the downstream process.
If the provider confirms failure, determine whether retrying is appropriate.
If the provider still cannot establish the state, move the transaction into a controlled recovery or investigation process.
The important idea is not that every provider supports exactly this workflow.
They don’t.
The principle is:
Don’t create another financial effect merely because communication failed.
Resolve the existing state wherever possible before creating a new one.
The Ledger Adds Another Layer of Protection
This is also where the concepts from payment processing begin connecting.
Transaction IDs solve identity.
Idempotency reduces duplicate business effects.
Transaction states model operational reality.
The ledger records financial effects.
Reconciliation compares your internal truth with external truth.
These aren’t five versions of the same mechanism.
They are controls against different failure modes.
Good architecture comes from understanding which problem each control solves.
What Production Systems Taught Me
Early in a project, architects naturally focus on the happy path.
Request.
Process.
Response.
Success.
Production teaches you to focus elsewhere.
What if the request arrives twice?
What if the callback arrives before another process finishes?
What if the provider succeeds and we don’t know?
What if our database succeeds and the next system fails?
What if two systems disagree for six hours?
The happy path tells you whether your feature works.
The failure paths tell you whether your architecture works.
That is a very different standard.
Design for Uncertainty
Distributed systems guarantee that uncertainty will eventually appear.
Networks fail.
Processes restart.
Messages arrive late.
Responses disappear.
Systems disagree.
Architecture cannot eliminate all of that.
But it can make uncertainty manageable.
Give important operations stable identities.
Make duplicate processing safe where possible.
Model intermediate and unknown states.
Build recovery paths.
Reconcile with external reality.
And never let a communication failure automatically convince you that a business operation failed.
Sometimes the most technically accurate answer is also the most uncomfortable:
We don’t know yet.
That isn’t weak architecture.
Pretending certainty exists when it doesn’t is weak architecture.
Good systems tell the truth, even when the truth is UNKNOWN.