Idempotency, state machines, and reconciliation for AI agents

Probabilistic brain. Deterministic muscles.

Trust is not a model property.

It is something you engineer around the model.

A lot of AI agent demos stop at the exciting part.

The model reasons.

It picks a tool.

The tool runs.

Green checkmark.

Done.

Production is usually where the boring questions arrive 😅

What happens if the request times out?

What happens if the action succeeded but the confirmation disappeared?

What happens if the agent retries?

What happens if your local state says one thing while the payment provider says something else?

This is where AI agents start looking suspiciously like distributed systems.

And that is a good thing.

Because distributed systems have spent decades dealing with exactly this kind of mess.

---

The $500 invoice that became $1,000

Imagine this workflow.

A user tells an agent:

Pay the $500 ACME invoice. The agent validates the invoice.

It calls the payment gateway.

The gateway successfully charges $500.

But the acknowledgement never makes it back.

Maybe the network drops.

Maybe the request times out.

Maybe the worker crashes at exactly the wrong moment.

The agent sees:

TIMEOUT

and concludes:

FAILED

So it tries again.

The second request also succeeds.

ACME receives $1,000.

Nice 😅

The interesting part is that nothing had to be obviously broken.

The LLM worked.

The tool call worked.

The gateway worked.

The retry worked.

Every individual component behaved in a way that looked reasonable.

The product still failed.

That is the difference between component success and system correctness.

The question is no longer:

Can the model call the payment tool?

It is:

Can the system make sure one business action stays one business action even when requests repeat?

That requires a few boring controls.

Boring controls are underrated.

The $500 invoice that became $1,000

---

Control 1: Idempotency Keys 🔐

The first problem is identity.

A retry creates another request.

But it should not necessarily create another business action.

Those are different things.

If the original intent is:

Pay invoice #84721 once. then the system needs a stable identity for that operation.

For example:

payment_request_id = 7f4e-91ba-22ac

The first payment request uses that ID.

If the request times out and the system retries, it uses the same ID.

Not a new one.

So the provider or your application can recognize:

I've already seen this operation. Instead of executing another payment, it returns the existing result.

That is the basic idea behind idempotency.

Repeated requests should not automatically create repeated side effects.

A useful mental model is:

Then the acknowledgement disappears.

Retry:

Not:

That last version is how the customer accidentally gets twice the money.

Or gets charged twice.

Or receives two orders.

Or gets two emails.

Or ends up with two tickets.

Or two account updates.

Same pattern.

Different invoice.

Control 1: Idempotency keys

---

The model should not generate the operation ID

This part matters.

Do not ask the model:

Create a unique payment ID. Then ask it again after a retry.

The model may happily produce a new value.

Now your retry looks like a completely new business operation.

The operation ID belongs to deterministic application state.

Not probabilistic model output.

The system should decide:

That ID persists until the operation reaches a resolved state.

The LLM may decide that a payment should be proposed.

The application owns the identity of that payment.

That separation is important.

Probabilistic brain. Deterministic muscles.

---

Control 2: Explicit State Machines 🧭

Identity solves one problem.

It does not solve everything.

The system also needs to know what state the workflow is actually in.

A lot of agent systems quietly rely on conversation history for this.

Something like:

We already tried paying the invoice earlier, so probably don't do it again. That is not enough.

The workflow needs explicit state.

For example:

Each state should have legal transitions.

The model can propose what happens next.

The state machine decides whether that transition is allowed.

That gives you a much safer boundary.

If the workflow is currently:

PAYMENT_PENDING

then:

SUBMIT PAYMENT

may be legal.

But if it is already:

PAYMENT_SUBMITTED

then the next action should probably not be:

SUBMIT PAYMENT AGAIN

It should be something closer to:

VERIFY PAYMENT STATUS

This sounds obvious when written down.

It becomes much less obvious inside a long agent trace with retries, tool errors and missing acknowledgements.

Explicit state makes the system easier to reason about.

---

The model proposes. The state machine decides.

This is the same architecture idea that keeps showing up in production AI.

The LLM is useful for reasoning under ambiguity.

It should not be the final authority over deterministic state transitions.

A safer pattern looks like:

Example:

The model might be wrong.

The product does not have to follow it.

That's the point.

Control 2: Explicit state machines

---

What about the timeout?

Now we get to the interesting state.

The payment request was submitted.

The provider may have completed it.

But the application did not receive confirmation.

What state are we in?

Not:

PAYMENT_FAILED

At least, not yet.

The better state is:

UNKNOWN

Or something more domain-specific like:

PAYMENT_STATUS_UNKNOWN

This distinction matters a lot.

Because:

unknown is not failed.

A timeout tells you that communication broke.

It does not necessarily tell you that the business operation failed.

The payment may have happened.

The email may have been sent.

The CRM may have updated.

The package may have shipped.

The acknowledgement may be the only thing that disappeared.

If your system collapses UNKNOWN into FAILED, retries become dangerous very quickly.

Unknown is not failed

---

Control 3: Reconciliation Loops 🔁

So what do we do with UNKNOWN?

We reconcile.

Reconciliation means comparing what your system believes happened with what actually happened in the external system.

For the payment example:

get_payment_status(payment_request_id)

Now we ask the provider:

What happened to operation 7f4e-91ba-22ac? Possible response:

SUCCESS

Great.

Update local state:

PAYMENT_SUBMITTED → PAYMENT_CONFIRMED

Do not pay again.

Or maybe the provider returns:

FAILED

Now you have evidence that the operation failed.

At that point, the system can decide whether another attempt is allowed.

Maybe the action is safe to retry.

Maybe it needs human review.

Maybe the error is permanent.

Maybe the workflow should recover another way.

The important part is that the retry happens after state becomes known, not because the application got nervous.

---

A safer failure flow

Instead of:

use:

This one change removes a surprising amount of chaos.

UNKNOWN is not an annoying edge case.

It is a real production state.

Model it.

Control 3: Reconciliation loops

---

Why all three controls matter 🧱

Idempotency, state machines and reconciliation solve related but different problems.

Idempotency asks:

What happens if the same command arrives twice? Its job is to prevent duplicate business actions.

State machines ask:

Is this action legal in the current state? Their job is to constrain workflow transitions.

Reconciliation asks:

Does our local state actually match reality? Its job is to repair ambiguity and drift between systems.

You usually need all three.

Idempotency without state can still leave the workflow confused.

State without idempotency can still allow duplicate external effects.

Both without reconciliation can leave you permanently stuck whenever acknowledgements disappear.

Together they turn ambiguous failures into predictable behaviour.

Why all three controls matter

---

The agent is not the source of truth

This is another useful distinction.

An agent might remember:

I think I already paid this. That is not a source of truth.

The conversation transcript is not a payment ledger.

The reasoning trace is not transactional state.

The model's memory is not your external system.

For consequential workflows, truth needs to live somewhere deterministic.

A database.

A payment provider.

An order system.

A durable workflow engine.

An event log.

Something the application can query and reconcile against.

The agent can interpret state.

It should not invent it.

---

A practical architecture

A production workflow might look like this:

Notice where the LLM sits.

Near the start.

It helps interpret intent and choose a proposed action.

It does not own:

• operation identity
• transaction state
• retry semantics
• legal transitions
• external truth
• reconciliation

That surrounding architecture is what makes the agent trustworthy.

---

Retry logic should not live only in the prompt

A prompt like:

Never process the same invoice twice. is useful.

Keep it.

But do not confuse it with the actual control.

If a duplicate request reaches the payment tool and your only protection is that sentence, the boundary is weak.

A stronger design says:

Now even if the model makes a bad decision, the system has another line of defense.

That's the pattern I care about in production AI.

Do not ask:

Can the model remember not to do this? Ask:

What stops the side effect if the model forgets?

---

Idempotency is not only for payments

Payments make the example obvious because duplicate money gets everyone's attention 😅

But the same problem exists anywhere the agent creates side effects.

Email

Ticketing

Orders

CRM

Provisioning

Notifications

Once an action changes the world, retry semantics matter.

---

Checkpoints should survive worker restarts

There is another failure mode hiding here.

Suppose your agent worker crashes.

Then restarts.

If workflow state only existed in memory, the new worker may have no idea what already happened.

That means this:

again.

Then the system runs the payment twice.

So consequential workflow checkpoints need to survive process restarts.

Persist them.

The exact technology depends on the system.

But the principle is simple:

A worker restart should not reset business reality.

---

Audit logs matter too

If something goes wrong, you should be able to reconstruct the operation.

Not the model's hidden reasoning.

The operational facts.

For example:

operation_id: 7f4e-91ba-22ac invoice_id: 84721 amount: 500 requested_at: 10:03:12 payment_submitted_at: 10:03:14 provider_timeout_at: 10:03:44 state_changed_to_unknown: 10:03:44 reconciliation_started: 10:03:46 provider_result: SUCCESS local_state_updated: PAYMENT_CONFIRMED duplicate_retry_blocked: true

That is useful evidence.

You can inspect what happened.

You can debug it.

You can explain it.

You can improve the workflow.

Reliability gets much easier when consequential tool calls leave an audit trail.

---

The trustworthy agent checklist 🛡️

Before letting an agent perform consequential writes, I would want to answer yes to these:

• Every consequential write supports idempotency.
• Workflow state is persisted outside the model.
• Invalid state transitions are blocked.
• UNKNOWN is modeled explicitly.
• Ambiguous failures reconcile against external reality.
• Checkpoints survive worker restarts.
• Consequential tool calls are auditable.
• Retry behaviour is bounded.
• Duplicate requests do not automatically create duplicate side effects.
• Operators can understand and recover unresolved state.

You do not need an enormous architecture on day one.

But if the agent can change real business state, these are no longer theoretical questions.

---

A small rule that prevents expensive bugs

When an external action times out, do not immediately ask:

Should we retry? First ask:

What state are we actually in? That usually leads to the right sequence:

Not:

The second one is much easier to implement.

The first one is much cheaper to operate.

---

Probabilistic brain. Deterministic muscles.

This is the architecture I keep coming back to.

Use the LLM where uncertainty and reasoning are useful.

Use deterministic application controls where correctness matters.

The model can propose:

Pay this invoice. The system should decide:

• which invoice
• which operation ID
• whether the transition is legal
• whether the action already happened
• whether retry is safe
• whether local state matches external reality
• what gets logged
• what happens when certainty disappears

That separation lets you get the leverage of probabilistic systems without making the rest of the product probabilistic too.

Probabilistic brain. Deterministic muscles.

---

Closing

You arrive for reasoning, agents and tool use.

You stay for UUIDs, queues, transactions, state machines, retries, audit logs and reconciliation 😅

That is not AI becoming less interesting.

That is AI becoming real engineering.

The model may be the most visible part of the system.

It is not the whole system.

Trust comes from what happens around it.

So when your agent hits a timeout, don't only ask:

What should the model do next? Ask:

What does the system actually know? Then make the next action from there.

Trust is an architectural invariant.

And for production agents:

The model can propose the action. The system has to own identity, state and recovery.