The Tool Timed Out. Did the Agent Just Do It Twice?

The customer asked for a callback.

The agent did exactly what it was supposed to do.

It verified the user, confirmed the request, and called the service tool.

Then the tool timed out.

No success response.

No failure response.

Just:

timeout

The agent had a decision to make.

Retry?

Or stop?

It retried.

The second call succeeded.

The customer received a confirmation:

Your callback request has been created.

Everything looked fine.

Until customer service found two requests.

The first tool call had actually succeeded.

The timeout happened after the request was created but before the result reached the agent.

The retry created a duplicate.

Nothing in the prompt was obviously wrong.

The real failure was in how the system handled uncertainty after a state-changing action.

That distinction matters.

A failed read is inconvenient.

An uncertain write can create real-world side effects.

A timeout does not always mean failure

When a read tool times out, retrying may be reasonable.

For example:

get_claim_status

If the first call fails, a second call can usually retrieve the same information without changing anything.

A write tool is different.

Consider:

create_callback_request

After a timeout, at least three states are possible:

1. Request was never created
2. Request was created successfully
3. Request is still processing

From the agent’s perspective, all three may look identical.

That means:

Timeout is not the same as failure.

It can mean unknown outcome.

And unknown outcomes need their own test cases.

The dangerous retry pattern

A naive agent loop often behaves like this:

Call tool
    ↓
Timeout?
    ↓
Retry
    ↓
Success

That works for many read operations.

For write operations, it can be dangerous.

A more realistic state machine is:

Call write tool
    ↓
Result received?
    ├─ Yes → Continue
    └─ No
        ↓
    Is outcome known?
        ├─ Yes, failed → Retry if allowed
        ├─ Yes, succeeded → Continue
        └─ Unknown → Reconcile before retry

The important branch is the last one.

If the agent does not know whether the side effect happened, retrying immediately may duplicate it.

A concrete example

Suppose the agent creates a manual review request.

The tool contract is:

name: create_review_request

inputs:
  claim_id: string
  reason: string

side_effect:
  Creates one review request.

success:
  status: created
  request_id: string

There is no idempotency key.

The agent calls:

{
  "claim_id": "CL-8211",
  "reason": "customer requested review"
}

The backend creates:

REV-101

Then the network connection breaks before the response returns.

The agent sees:

timeout

It calls the tool again.

The backend creates:

REV-102

Now two valid review requests exist.

The second call did not fail technically.

The first call did not fail technically either.

The failure was in the distributed interaction between them.

This is a system test, not a prompt test

You can add a prompt instruction:

Do not retry write actions after a timeout.

That may help.

But it is a weak place to enforce a hard operational guarantee.

A production system should encode retry safety in the tool contract and execution layer.

For example:

name: create_review_request

inputs:
  claim_id: string
  reason: string
  idempotency_key: string

retry:
  allowed_only_with_same_idempotency_key

Now both attempts can use:

idempotency_key = REVIEW-CL-8211-20260812-001

If the first call created the request, the second call returns the same transaction instead of creating another.

That turns an uncertain retry into a safe retry.

What idempotency actually gives you

An idempotent write means that repeating the same logical operation does not create additional side effects.

For example:

Attempt 1:
create callback
key = CB-441
→ request created: SR-77192

Attempt 2:
create callback
key = CB-441
→ existing request returned: SR-77192

The result is still one callback request.

This is far safer than relying on the model to remember whether it already tried.

But idempotency is not magic

Idempotency keys must be designed carefully.

A bad key can make two genuinely different actions look identical.

Suppose a customer asks for two separate callbacks on the same claim.

If the key is simply:

claim_id = CL-8211

the second legitimate request may be blocked.

A better key represents one intended transaction:

callback-request:
customer=C-18427
claim=CL-8211
intent_instance=REQ-9917

The system needs a stable identifier for the action, not merely the entity.

Test the uncertain state explicitly

Many test suites include:

• Success
• Validation failure
• Forbidden
• Timeout
• Service unavailable

But they treat timeout as simple failure.

For write operations, add a separate state:

outcome_uncertain

Then define the expected behaviour.

Example test:

scenario: write_timeout_after_commit

tool:
  create_callback_request

backend_behavior:
  side_effect_occurs: true
  response_returns: false

expected:
  automatic_retry: false
  duplicate_side_effects: 0
  status: uncertain
  reconciliation_attempted: true
  user_told_success: false
  user_told_failure: false

This tests the system much more accurately.

Reconciliation is the missing step

If the write outcome is uncertain, the safest next step is usually not retry.

It is reconciliation.

That means asking:

Did the action happen?

For example:

create_callback_request
        ↓
timeout
        ↓
get_callback_request_by_idempotency_key
        ↓
found?
   ├─ Yes → Treat as completed
   └─ No  → Retry if policy allows

The system may reconcile using:

• Idempotency key
• Transaction ID
• Request fingerprint
• External status endpoint
• Event log
• Audit record
• Manual review

The exact method depends on the external system.

The principle is stable:

Resolve uncertainty before creating another side effect.

What should the agent tell the user?

This matters too.

A weak response after timeout is:

The request failed. I’ll try again.

That assumes failure.

Another weak response is:

Your callback has been created.

That assumes success.

A safer response is:

I submitted the callback request, but I could not confirm whether it completed. I’m checking the status before trying again.

If reconciliation is unavailable:

I could not confirm whether the request completed. I will not submit it again automatically because that could create a duplicate. The request needs to be checked before another attempt.

The language should reflect the actual system state.

Unknown should stay unknown.

The trace should prove what happened

Now imagine investigating the duplicate.

A weak trace might show:

10:14:12 create_callback_request → timeout
10:14:15 create_callback_request → success

That is not enough.

You need to know whether both calls represented the same logical action.

A stronger trace contains:

run_id: RUN-9811
logical_action_id: ACTION-441

attempt_1:
  tool_call_id: TOOL-100
  idempotency_key: CB-441
  result: timeout
  side_effect_state: uncertain

attempt_2:
  tool_call_id: TOOL-101
  idempotency_key: CB-441
  result: existing_request
  transaction_id: SR-77192

Now the team can reconstruct the interaction.

Observability is part of retry safety.

Test more than the happy timeout

There are several different timeout scenarios worth testing.

Timeout before processing

The backend receives nothing.

Expected:

• Safe retry may be allowed.

Timeout during processing

The backend state is unknown.

Expected:

• Reconcile before retry.

Timeout after commit

The action succeeded but the response was lost.

Expected:

• Do not duplicate.
• Reconcile.
• Return the existing transaction.

Response arrives after the agent gives up

The tool completes after the workflow has timed out.

Expected:

• Late result is correlated correctly.
• External state is not lost.
• The user is not sent contradictory updates.

Retry itself times out

Now the uncertainty grows.

Expected:

• Stop uncontrolled retries.
• Escalate or reconcile through another path.

Add partial success

Real write operations often span multiple systems.

Suppose an address change updates:

Customer profile: success
Home policy: success
Travel policy: timeout
Life policy: unavailable

A single response such as:

status = failed

is misleading.

So is:

status = completed

A better model uses explicit composite states:

requested
in_progress
partially_completed
completed
failed
uncertain

Now the agent can report:

Your customer profile and home policy were updated. The travel-policy update could not be confirmed, and the life-policy system is unavailable.

That is operationally accurate.

Before: the naive system

Suppose we create 30 write-action test cases covering:

• Normal success
• Validation failure
• Timeout
• Timeout after commit
• Duplicate message
• Duplicate agent attempt
• Partial backend completion
• Unknown transaction state

The original agent uses simple retry logic.

Results:

30 scenarios

Task completion:              93%
User-visible success:         90%
Duplicate side effects:        4
Uncertain states misreported:  6
Unsafe retries:                7

At first glance, 93 per cent task completion looks respectable.

Operationally, four duplicate actions are unacceptable.

Changes

We introduce:

• Stable logical action IDs
• Idempotency keys
• Explicit uncertain states
• Retry policies per tool
• Reconciliation before repeat writes
• Transaction lookup
• Duplicate detection
• Structured retry traces
• User messages for unresolved outcomes

We also separate tool types into:

READ
WRITE_IDEMPOTENT
WRITE_NON_IDEMPOTENT

Retry policy is no longer generic.

After: the controlled system

We run the same test set again.

30 scenarios

Task completion:              97%
User-visible success:         93%
Duplicate side effects:        0
Uncertain states misreported:  0
Unsafe retries:                0

Notice something interesting.

User-visible success did not become 100 per cent.

That is fine.

In several scenarios, the correct behaviour is to report uncertainty or escalate.

A reliable system is not one that always returns success.

It is one that represents the real state honestly and avoids making things worse.

Deterministic assertions are powerful here

Write safety is an area where ordinary software assertions are extremely valuable.

You can verify:

transaction_count <= 1

or:

retry_requires_same_idempotency_key = true

or:

unknown_outcome → no automatic second write

or:

authorization = allowed
AND
confirmation = present
AND
transaction_id != null

These checks should not be left entirely to an LLM judge.

Use models to evaluate natural-language behaviour.

Use deterministic controls for transactional guarantees.

Multi-agent systems make this harder

The risk grows when several agents can perform the same action.

Imagine:

• Claims worker decides a review request is needed.
• Coordinator independently reaches the same conclusion.
• Escalation worker also creates a request.

All three are individually reasonable.

Together they create duplicates.

The architecture should therefore define one write authority.

For example:

Workers:
recommend actions

Coordinator:
approves workflow state

Write-action service:
executes once

A shared idempotency key should follow the logical action across agents.

Do not give every agent independent write authority unless the use case genuinely requires it.

Make retry behaviour part of the tool contract

A tool contract should document:

• Is it read-only?
• Does it create side effects?
• Is it idempotent?
• May it be retried?
• Which errors are retryable?
• Which idempotency key is required?
• How is status reconciled?
• How are partial results represented?

Example:

tool: create_callback_request

type: write

idempotency:
  required: true

retry:
  timeout: reconcile_first
  unavailable: no
  rate_limited: yes
  validation_error: no

status_lookup:
  supported: true

side_effect:
  creates_service_request

This is far more useful than:

Creates a callback request.

Every write tool needs failure-injection tests

Do not wait for production networking to fail.

Inject faults deliberately.

Test:

• Lost response
• Delayed response
• Duplicate response
• Partial commit
• Retry after uncertain state
• Backend success followed by gateway error
• Out-of-order acknowledgement
• Tool result malformed after commit

Then inspect:

• Number of side effects
• User-visible status
• Retry behaviour
• Trace completeness
• Transaction correlation

This turns an ugly production incident into a repeatable test.

FAQ

Should an AI agent retry after a tool timeout?

It depends on the tool. Retrying a read-only operation is often safe. Retrying a state-changing operation may create duplicate side effects. For writes, the system should understand whether the outcome is known and reconcile uncertain transactions before retrying.

What is idempotency in AI agent tools?

Idempotency means that repeating the same logical write does not create additional side effects. An idempotency key allows repeated calls to return the original result instead of creating duplicate transactions.

Why is a timeout different from failure?

A timeout only tells the caller that a response did not arrive in time. The backend may have failed, succeeded, or still be processing. That makes the transaction state uncertain.

Should retry logic live in the prompt?

Prompt instructions can help, but transactional safety should be enforced in application code, tool contracts, workflow state, and external systems whenever possible.

How do you test duplicate actions?

Inject response loss or timeout after the backend commits a write, then verify that the agent does not create a second transaction. Assert on transaction count, idempotency keys, reconciliation, and user-visible status.

Test the side effect, not only the conversation

It is easy to evaluate an agent by reading the final response.

In this case, the final response looked perfect:

Your callback request has been created.

The real defect lived outside the conversation.

There were two callbacks.

That is why production agent testing has to include:

• Tool traces
• External state
• Transaction IDs
• Retry behaviour
• Idempotency
• Reconciliation
• Partial completion
• User communication

The final sentence is only one output of the system.

A practical test you can add today

Pick one write tool used by your agent.

Create three cases:

1. The write succeeds normally.
2. The write fails before execution.
3. The write succeeds, but the response is lost.

For each one, define:

• Whether retry is allowed
• Required idempotency behaviour
• External transaction count
• Expected user-visible status
• Required trace evidence
• Escalation path

The third case is the important one.

If your expected behaviour is simply:

Retry the tool

you may have found your next production incident before your customers do.

Testing complete agent systems

This is one of the themes behind The AI Agent Test Manual.

AI agent quality is not only about whether the model produces a good response.

It is also about whether the surrounding system handles tools, permissions, state, retries, retrieval, memory, partial failure, observability, and real-world side effects safely.

The book covers those topics with practical test strategies, examples, evaluation pipelines, incident patterns, and release controls.

The AI Agent Test Manual:
https://amzn.eu/d/0blIkveq

The key lesson is simple:

Never let “I did not receive a response” silently become “the action did not happen.”

For write operations, uncertainty is a state.

Test it.