Token LabSearch

Reliable agent actions: timeouts, retries and recovery

Build an agent execution boundary that survives lost acknowledgements and restarts. Follow an e-commerce refund from approved intent to reconciled outcome, with durable state, bounded retries and a runnable local reference.

Sources & version details
In this article

1. The refund that succeeded after the agent gave up

A customer requests a $25 refund for order 7. The model proposes an action, a policy service approves it, and the payment provider accepts it. The response disappears before the agent stores success. A naïve agent reports failure and creates another refund. The model may have reasoned correctly both times; the execution boundary still caused a business error. Our objective is to preserve one approved business intent across uncertain delivery, not merely to make the conversation look successful.

A side effect changes an external system. An acknowledgement is evidence delivered to the caller; its absence does not prove absence of the effect. Idempotency is a contract under which repeated submissions of the same identified intent do not create an additional effect. Durability means state survives a specified failure model. These concepts are independent: a durable log can record two non-idempotent payments perfectly. Prerequisites are database transactions, request/response communication and access control. A transaction protects operations in its own database; it does not automatically include a remote HTTP call.

This tutorial is for agents that create tickets, reserve inventory or initiate approved refunds. Read-only search needs freshness and retry budgets but usually not payment-style effect reconciliation. Non-reversible, non-idempotent actions without a status API require an operator or a redesigned integration. The worked system is an original reference design. The downloadable program uses two local databases and no real payment or model API.

2. Separate planning authority from execution authority

The planner produces a proposed tool name and arguments. A deterministic gate validates schema, tenant, order ownership, amount and approval expiry. Normalize money to integer minor units before approval: 2500 cents is $25, with the currency recorded separately in a real integration. Bind the approval to an operation ID, canonical argument digest and policy version. Changing the amount after approval creates a new intent that must be reviewed; retrying the same intent retains the original ID.

The executor durably records intent before sending it. The provider owns the authoritative effect record. A reconciler queries that provider and advances local state only from validated evidence. The user interface distinguishes pending, confirmed, uncertain and rejected. Tool output is untrusted data: a provider response cannot grant a wider scope or authorize the agent's next purchase. A retry gate must recheck authorization for a new outbound mutation; permission to inspect the status of a previous operation is a separate capability.

Use two boundaries in the architecture: the local database transaction and the remote provider transaction. There is a network gap between them. An outbox can atomically save local intent and dispatch work in the local database, then deliver at least once; it cannot make the provider's effect atomic with that transaction. Every arrow carrying a mutation passes through the execution gate. Reconciliation only reads remote status and records evidence, never silently issues compensation.

Original reference architecture: solid arrows show command/result flow; dashed arrows show durable records or status reconciliation. The two database transactions are independent.
Original reference architecture: solid arrows show command/result flow; dashed arrows show durable records or status reconciliation. The two database transactions are independent. Open full-size diagram

3. Model uncertainty as a state, not an exception message

Persist PREPARED before sending. SUBMITTED means a request may have escaped; it is not proof of acceptance. A confirmed matching result allows SUCCEEDED. A lost acknowledgement or crash after submission leads to UNKNOWN or a SUBMITTED record requiring reconciliation. REJECTED is appropriate only for definitive rejection; a timeout is not that evidence. If the provider reports not found, consider visibility lag and an in-flight request. Only a documented idempotency contract makes repeating the same key safe in this situation.

Consider three crash windows. Before local intent commits, no send is allowed. After intent commits but before the send, recovery can resume the same operation. After the provider commits but before local success commits, recovery must ask the provider or repeat the same deduplicated operation. Generating a new UUID on every retry breaks the identity across that last window. Derive a stable key from an existing business operation or persist a generated key in a completed durable step before dispatch.

Cancellation stops future work where possible; it does not reverse a completed effect. If cancellation races with a refund commit, show the reconciled outcome and evaluate any remedy as a new authorized operation. A compensation is a business action with its own failure modes, not an atomic rollback. LangGraph's current fault-tolerance documentation similarly distinguishes retryable node execution from external effects and treats missing acknowledgement as unknown. Pin framework versions when implementing; our reference avoids framework-version dependencies.

4. Idempotency needs an owner, scope and retention period

A useful provider record is keyed by tenant and operation ID, and stores the canonical intent digest, terminal result and retention policy. The provider must make deduplication and its effect atomic within its own boundary. A client-side lookup followed by an unprotected write is insufficient: two workers can both observe no result and both act. Unique keys plus a transaction protect the local reference; a production provider must publish its equivalent contract.

Do not hash raw prompts or use a model-generated call ID as the sole business identity. Harmless whitespace can change raw hashes, while two intentionally distinct identical refunds must remain distinguishable. Canonicalize validated arguments, but keep operation identity separate from the digest. Our example hashes order ID and integer cents only; production must include currency, account, action type and schema version. A different digest under the same key is a conflict, not a reason to overwrite history.

Retention creates a real limit. Stripe API v1 documents replayed responses, parameter mismatch checks and key pruning after at least 24 hours; API v2 has different failed-operation behavior. This is a provider-specific example, not a universal guarantee. If your recovery horizon exceeds deduplication retention, stop automatic mutation until you reconcile using a durable business identifier. Preserve a compact tombstone or business ledger when necessary. Idempotency keys must not expose sensitive personal information.

5. Deadlines and retry budgets prevent recovery from becoming overload

A timeout bounds one wait; a deadline bounds the entire operation. Let remaining budget R equal deadline D minus current time t. Reserve F seconds for final status recording and response. A new attempt with timeout T and chosen backoff b fits only when b+T+F does not exceed R. In an original 30-second example, 18 seconds have elapsed, F=2 and T=4: an 8-second backoff would need 14 seconds, so it must not start. A 2-second backoff needs 8 seconds and fits the 12-second remainder. Use monotonic time within a process and propagate bounded remaining time across services.

Classify before retrying. Invalid arguments, denied permissions and definitive business rejections are not transient transport errors. For temporary errors use a capped attempt count, exponential backoff with jitter, a shared retry budget and dependency concurrency limits. The formula below defines full jitter: attempt index i starts at zero, base and cap are seconds, and the random delay is uniform between zero and the cap for that attempt. Jitter spreads retries; it does not reduce the total number of permitted attempts.

Nested policies multiply work. Three attempts at each of three layers can produce 27 leaf calls in the worst case; choose one owner for retry policy or explicitly budget the combined tree. Under an illustrative independent failure probability p=0.2 and at most three attempts, expected attempts are 1+0.2+0.04=1.24. Correlated overload violates that independence assumption. A retry budget controls amplification even when the probabilistic estimate fails. gRPC deadline propagation helps stop useless downstream work, but application-spawned work must cooperate with cancellation.

In these equations, b_0 is base delay, b_max is the delay cap, i is the retry index, A is the number of attempts, m is the maximum allowed attempts and p is the independent per-attempt failure probability. A and m are counts; p is dimensionless. The expectation includes the first attempt.

6. Durable execution and observability solve different problems

A graph checkpoint remembers control state. A tool ledger records identified effects. An audit trail records who authorized what and which evidence justified the outcome. A trace measures attempts, queues and latency. None replaces the others. LangGraph distinguishes thread checkpoints from cross-thread stores; in-memory examples are useful during development but do not establish crash durability. Persist model outputs or decisions at replay boundaries so recovery does not ask a stochastic planner to invent a different operation midway through the same approved intent.

Store operation ID, tenant, intent digest, approval version, attempt ID, deadline, provider reference, state transition and a redacted error class. Keep attempt identity separate: one business operation can have many network attempts. Measure unresolved-UNKNOWN count and age, effects per approved intent, conflict rate, retry amplification and time to reconciled outcome. Do not count a timed-out request as a failed refund if the provider later confirms it. Protect logs and retention because prompts and tool payloads can contain personal data.

For multiple executors, use leases for work ownership and a fencing generation where the destination supports it. A lease expiring does not kill an old worker; without fencing or provider idempotency it can still act. Avoid placing a local mutex in a diagram and calling it distributed exclusion. Our code deliberately uses one executor and a transactional provider model; it does not implement distributed leases, cross-region failover or a real authorization service.

7. Choose the smallest recovery mechanism that preserves the contract

For a read-only lookup, retry within the deadline and label stale data. For an idempotent mutation with status lookup, reconcile first and safely resubmit the same key under valid authorization. For a mutation with deduplication but no query, a repeated request can recover a stored response only within the provider's documented retention and error semantics. For a non-idempotent action with no reliable status, stop and request operator investigation. Blind automatic retries cannot manufacture a missing contract.

A workflow engine is valuable when many long-running steps, timers and human approvals require durable orchestration. It is excessive for a single local pure computation. A transactional outbox is useful for reliable local dispatch, while a saga specifies business compensation across completed steps. Neither makes arbitrary third-party APIs exactly-once. For the refund case, a status-only reconciler is safer than letting the planner infer success from a customer email or immediately issue a reverse payment.

Remaining engineering problems include deduplication expiry, provider outages during reconciliation, migrations of stored state schemas, late callbacks, reordered events and multi-region ownership. Version the intent schema, retain evidence of every terminal transition and define a manual queue with an accountable owner. The production acceptance criterion is a documented safe behavior for each uncertainty window, not a claim that the agent never fails. Practice below verifies a small subset; it is not certification of a payment integration.

MechanismProtectsDoes not guarantee
CheckpointWorkflow stateExactly-once external effects
OutboxLocal intent + dispatch recordAtomic commit across providers
Idempotency keySame-intent deduplicationUnlimited retention or authorization
ReconciliationEvidence of remote outcomeAvailability of the remote API

Practice and validation

Python 3.11+, standard library only. Run python3 agent-action-recovery.py. No API keys, network, external effects or random seed. Two temporary SQLite files deliberately separate agent state from the simulated provider. The provider commits deduplication and its simulated refund together; no transaction spans both files. Expected output: PASS followed by effects=3; duplicate-effects=0. The three effects are two distinct shop operations and one operation in another tenant. Assertions cover lost acknowledgement, reopening client state, changed-argument conflict, expired write permission, the provider-commit/client-commit crash window and duplicate delivery. This checks deterministic recovery logic, not real IAM, process-kill durability, network behavior or refund limits. Add a baseline that submits a fresh operation ID after lost acknowledgement: observe the extra effect, then restore the stable ID. Keep failure positions and business inputs identical. Extend with delayed status visibility, provider errors, budget exhaustion and concurrent executors before production.

agent-action-recovery.py
python3 agent-action-recovery.py
Python 3.11+
"""Python 3.11+, standard library. Local teaching model; never contacts a provider."""
import hashlib
import json
import sqlite3
import tempfile
from pathlib import Path


def connect(path):
    db = sqlite3.connect(path)
    db.execute('PRAGMA synchronous=FULL')
    return db


def fingerprint(order, cents):
    if type(cents) is not int or cents <= 0:
        raise ValueError('positive integer cents required')
    return hashlib.sha256(json.dumps([order, cents], separators=(',', ':')).encode()).hexdigest()


class Provider:
    """One transactional SQLite database models the external system's contract."""
    def __init__(self, path):
        self.db = connect(path)
        self.db.executescript('''
        CREATE TABLE IF NOT EXISTS refunds (
          tenant TEXT, op TEXT, digest TEXT, cents INTEGER,
          PRIMARY KEY (tenant, op));
        ''')

    def submit(self, tenant, op, digest, cents):
        # Atomic deduplication + effect, not a client-side check-then-act.
        with self.db:
            self.db.execute('BEGIN IMMEDIATE')
            old = self.db.execute('SELECT digest FROM refunds WHERE tenant=? AND op=?',
                                  (tenant, op)).fetchone()
            if old and old[0] != digest:
                raise ValueError('provider key conflict')
            if not old:
                self.db.execute('INSERT INTO refunds VALUES (?,?,?,?)',
                                (tenant, op, digest, cents))
        return {'state': 'succeeded', 'digest': digest}

    def status(self, tenant, op):
        row = self.db.execute('SELECT digest FROM refunds WHERE tenant=? AND op=?',
                              (tenant, op)).fetchone()
        return None if row is None else {'state': 'succeeded', 'digest': row[0]}

    def count(self):
        return self.db.execute('SELECT COUNT(*) FROM refunds').fetchone()[0]


class AgentExecutor:
    def __init__(self, path, provider):
        self.db, self.provider = connect(path), provider
        self.db.execute('''CREATE TABLE IF NOT EXISTS operations (
          tenant TEXT, op TEXT, digest TEXT, order_id TEXT, cents INTEGER,
          state TEXT, PRIMARY KEY(tenant,op))''')
        self.db.commit()

    def state(self, tenant, op):
        row = self.db.execute('SELECT state FROM operations WHERE tenant=? AND op=?',
                              (tenant, op)).fetchone()
        return row[0] if row else None

    def mark(self, tenant, op, state):
        with self.db:
            self.db.execute('UPDATE operations SET state=? WHERE tenant=? AND op=?',
                            (state, tenant, op))

    def execute(self, tenant, op, order, cents, *, authorized, lose_ack=False,
                crash_after_provider=False):
        digest = fingerprint(order, cents)
        row = self.db.execute('SELECT digest,state FROM operations WHERE tenant=? AND op=?',
                              (tenant, op)).fetchone()
        if row and row[0] != digest:
            raise ValueError('same operation ID with different intent')
        if row and row[1] == 'succeeded':
            return 'succeeded'
        if not row:
            if not authorized:
                return 'blocked'
            with self.db:
                self.db.execute('INSERT INTO operations VALUES (?,?,?,?,?,?)',
                                (tenant, op, digest, order, cents, 'prepared'))
        # Reconcile first, even if permission to create a new effect has expired.
        # This demo assumes a separate permission to inspect this tenant's status.
        result = self.provider.status(tenant, op)
        if result:
            if result['digest'] != digest:
                raise ValueError('status intent mismatch')
            self.mark(tenant, op, 'succeeded')
            return 'succeeded'
        if not authorized:
            return 'blocked'
        # A missing status alone is NOT proof of no effect. Safe repeat here depends
        # on Provider.submit's durable atomic idempotency contract, with no expiry.
        self.mark(tenant, op, 'submitted')
        self.provider.submit(tenant, op, digest, cents)
        if crash_after_provider:
            raise RuntimeError('injected crash after provider commit')
        if lose_ack:
            self.mark(tenant, op, 'unknown')
            return 'unknown'
        self.mark(tenant, op, 'succeeded')
        return 'succeeded'


def run():
    with tempfile.TemporaryDirectory() as folder:
        root = Path(folder)
        provider = Provider(root / 'provider.sqlite')
        client = AgentExecutor(root / 'agent.sqlite', provider)
        assert client.execute('shop', 'refund-1', 'order-7', 2500,
                              authorized=True, lose_ack=True) == 'unknown'
        assert provider.count() == 1
        client.db.close()
        client = AgentExecutor(root / 'agent.sqlite', provider)
        assert client.execute('shop', 'refund-1', 'order-7', 2500,
                              authorized=False) == 'succeeded'
        assert provider.count() == 1
        try:
            client.execute('shop', 'refund-1', 'order-7', 3000, authorized=True)
            raise AssertionError('changed intent accepted')
        except ValueError:
            pass
        assert client.execute('shop', 'refund-2', 'order-8', 500,
                              authorized=False) == 'blocked'
        try:
            client.execute('shop', 'refund-3', 'order-9', 700,
                           authorized=True, crash_after_provider=True)
        except RuntimeError:
            pass
        assert client.state('shop', 'refund-3') == 'submitted'
        client.db.close()
        client = AgentExecutor(root / 'agent.sqlite', provider)
        assert client.execute('shop', 'refund-3', 'order-9', 700,
                              authorized=True) == 'succeeded'
        # Direct duplicate delivery exercises provider-side atomic deduplication.
        d = fingerprint('order-9', 700)
        provider.submit('shop', 'refund-3', d, 700)
        assert provider.count() == 2
        # Distinct tenant namespace: same op is a different operation.
        assert client.execute('shop-2', 'refund-3', 'order-9', 700,
                              authorized=True) == 'succeeded'
        assert provider.count() == 3
        client.db.close()
        provider.db.close()
    print('PASS: lost acknowledgement, restart, conflict, expired permission, crash window, duplicate delivery, tenant namespace')
    print('effects=3; duplicate-effects=0; no network; deterministic; no random seed needed')


if __name__ == '__main__':
    run()

Original interview exercises—not company questions

Why is a timeout not proof that a refund failed?

Hint

Separate effect commit from response delivery.

Answer and analysis

The provider can commit before the response is lost. Keep an uncertain state and reconcile; do not create another business intent.

Can a graph checkpoint guarantee exactly-once payments?

Hint

Locate the two transaction boundaries.

Answer and analysis

No. Recovery can rerun a node after an external effect. The provider needs a documented atomic idempotency contract or a reliable reconciliation procedure.

May an expired approval block reconciliation?

Hint

Separate read authority from mutation authority.

Answer and analysis

It should block new writes. Status reads may proceed under a separate valid scope; the demo explicitly assumes that scope. Do not let a status response refresh approval.

When can a repeated operation become unsafe again?

Hint

Inspect retention, arguments and ownership.

Answer and analysis

After deduplication expiry, under a different key, with a changed intent, or outside the contracted namespace. Stop and reconcile rather than silently generating a replacement key.

Scoring: 1—defines timeout/idempotency; 2—identifies uncertainty; 3—runs and explains recovery; 4—designs failure controls; 5—specifies retention, authorization and multi-worker boundaries.

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope

References and further learning

Continue: agent scheduling