The Agent Was Confident. The Data Was Wrong.

The most dangerous AI agent failures are not always the ones that crash.

Sometimes the tool works. Retrieval returns documents. The model responds fluently. The answer sounds entirely reasonable.

And it is wrong.

Consider this customer question:

Is my freestanding garden shed covered for storm damage?

The agent searches the company knowledge base and finds a document that says:

Detached structures on the insured property are covered for storm damage.

It answers:

Yes. Your garden shed is covered for storm damage under your home insurance.

Clear. Helpful. Confident.

There is just one problem.

The retrieved document is from last year.

The current policy wording contains an additional condition: freestanding lightweight structures must be permanently anchored to qualify for storm-damage coverage.

The agent did not hallucinate in the usual sense. It did not invent a policy. It answered from retrieved evidence.

The evidence itself was wrong for the question.

This is one of the easiest agent failures to miss because almost every component appears to have worked.

The retrieval call succeeded.

The document existed.

The model used the document.

The answer was grounded in the supplied context.

A simple grounding judge might even mark it as correct.

That is why retrieval quality needs to be tested as its own system.

Retrieval failure is different from a tool failure

Tool failures are often obvious.

An API may return:

timeout

or:

403 Forbidden

or:

customer_not_found

Your application can detect those states.

Retrieval failures are often quieter.

The search system may return five perfectly plausible passages. They may even be about exactly the right subject.

But they can still be:

• Outdated
• Incomplete
• From the wrong product
• Lower-authority guidance
• A draft
• A training example
• Missing an important exception
• Correct generally but wrong for this customer

Nothing necessarily throws an exception.

Instead, bad evidence flows smoothly into the model.

That is what makes this class of failure dangerous.

Grounded does not necessarily mean correct

Teams often add retrieval-augmented generation, or RAG, to reduce hallucinations.

The mental model is sensible:

Retrieve trusted information, give it to the model, and require the answer to stay grounded in that information.

But this shifts part of the quality problem from generation to retrieval.

If you retrieve the wrong evidence, the model can produce a beautifully grounded wrong answer.

Imagine this evidence package:

Question:
Is my freestanding garden shed covered for storm damage?

Retrieved passage:
Detached structures on the insured property are covered for storm damage.

Agent answer:
Yes. Your garden shed is covered for storm damage.

A grounding evaluator asks:

Is the answer supported by the retrieved passage?

It may correctly answer:

Yes.

The evaluator did its job.

The retrieval system did not.

The missing question is:

Did we retrieve the evidence that should have been used?

That needs a different test.

Treat retrieval as its own test surface

A retrieval evaluation should run before the final answer is generated.

Instead of asking only whether the final response is good, define what evidence the query should produce.

For our garden-shed example, the test fixture could look like this:

query: "Is my freestanding garden shed covered for storm damage?"

required_evidence:
  - current detached-structures coverage rule
  - permanent-anchoring condition

forbidden_or_misleading:
  - expired policy version
  - training example
  - product brochure without the exception

requirements:
  current_policy_ranked_first: true
  permissions_respected: true

Now we can test the search system independently from the model.

That distinction matters.

If the retrieval test fails, changing the prompt is probably not the right fix.

Test more than one wording

One query is not enough.

Customers rarely use policy terminology.

The same question might arrive as:

• Is my garden shed insured against storms?
• What happens if wind damages the shed behind my house?
• Does home insurance cover a shed that is not fixed to the ground?
• The storm moved my lightweight shed. Is that covered?
• Is storm damage to an outdoor shed included?

These should retrieve substantially the same authoritative evidence.

A good test set therefore contains variations of the same underlying intent.

This catches a common failure pattern: retrieval works for the language used in the documentation but fails for the language customers actually use.

Test freshness explicitly

Freshness deserves its own assertions.

Suppose your knowledge base contains:

HOME-POLICY-2025
HOME-POLICY-2026

Both mention detached structures.

The 2025 policy ranks slightly higher because its wording matches the customer query more closely.

Semantically, the search result looks excellent.

Operationally, it is wrong.

Useful assertions include:

• Current approved versions rank above expired versions.
• Future policy versions are not applied before their effective date.
• Draft documents do not appear as authoritative evidence.
• Cached results do not survive beyond their allowed lifetime.
• Effective dates are available to the retrieval layer.
• The version used in the answer is traceable.

This is not just a search-quality problem.

It is a production correctness problem.

Test whether the exception is retrieved

Many business rules are not contained in a single paragraph.

The answer may require:

1. A general rule
2. An exception
3. Customer-specific information

That creates another quiet failure.

Suppose retrieval returns:

Detached structures are covered for storm damage.

But not:

Lightweight freestanding structures must be permanently anchored.

The result is relevant.

It is simply incomplete.

So define a minimum evidence set:

Required evidence:

- Coverage rule
- Applicable exception
- Customer policy version

If any element is missing:

- Do not provide a definitive coverage answer.
- Ask for clarification or state that coverage cannot yet be verified.

Now the system has an evidence gate.

The model is allowed to answer confidently only when the required evidence exists.

Add conflict tests

Real knowledge bases are messy.

You may have:

• Current policy terms
• Previous policy terms
• FAQs
• Internal guidance
• Training documents
• Drafts
• Marketing copy

Eventually, some of them will disagree.

Your evaluation suite should deliberately test that situation.

For example:

Source A:
Detached structures are covered.

Source B:
Freestanding structures require permanent anchoring.

Source C:
All garden structures are covered automatically.

Source C is an outdated training example.

The correct behaviour is not to average the three statements.

The system needs a source hierarchy.

For example:

Customer-specific contract
>
Current approved policy terms
>
Current approved operational guidance
>
Public information
>
Training material
>
Drafts and examples

Better still, enforce as much of this hierarchy as possible in metadata and retrieval rather than hoping the language model resolves it correctly every time.

A worked before-and-after evaluation

Here is a simplified example of how I would measure the change.

Assume we create 40 retrieval cases covering:

• Query variations
• Current versus outdated policies
• Rule-plus-exception questions
• Conflicting sources
• Product boundaries
• Missing evidence
• Permission restrictions

Before

The original system is evaluated only on whether the final answers sound correct and are grounded in retrieved context.

Results:

40 test cases

Final-answer pass rate:        90%
Grounding pass rate:           95%

Current-version retrieval:     82%
Required-evidence retrieval:   75%
Conflict handling:             70%

At first glance, the agent looks good.

Ninety per cent of answers pass.

But retrieval-specific testing exposes the real weakness.

One quarter of the cases do not retrieve the complete evidence required for a reliable answer.

Changes

We introduce:

• Document-status metadata
• Effective-date filtering
• Source-priority rules
• Required-evidence fixtures
• Query-variation tests
• Conflict cases
• Retrieval regression tests
• A deterministic evidence gate before definitive answers

After

We run the same evaluation set again:

40 test cases

Final-answer pass rate:        98%
Grounding pass rate:           98%

Current-version retrieval:    100%
Required-evidence retrieval:   98%
Conflict handling:             95%

The important improvement is not merely that the final-answer score increased from 90 to 98 per cent.

We now know why we should have more confidence in the answer.

The correct sources arrive more reliably.

The necessary exceptions are present.

Old versions are controlled.

Conflicts have defined handling.

And when the evidence is incomplete, the system knows that it should not guess.

That is a much stronger quality claim than:

The answer looked good in our test conversations.

Retrieval failures should become regressions

Once you find a failure, keep it.

Suppose production exposes this query:

The wind tipped over the little shed in my garden. Does insurance pay for that?

The system retrieves the general storm-damage rule but misses the anchoring condition.

After fixing the problem, add the exact case—or a safely sanitised equivalent—to the regression suite.

Now future changes to:

• Embeddings
• Search models
• Chunking
• Metadata
• Ranking
• Query rewriting
• Documents
• Filters

must survive that test.

Retrieval evaluation should run when retrieval behaviour can change, not just when the prompt changes.

What should you measure?

You do not need twenty metrics on day one.

Start with the ones that tell you whether the correct evidence reaches the model.

I would begin with:

Required-evidence recall

Did retrieval return all evidence needed to answer the question?

Current-version accuracy

Did it prefer the currently applicable source?

Source-authority accuracy

Did higher-authority evidence outrank training material, marketing content, or drafts?

Conflict-handling accuracy

Did the system recognise conflicting sources instead of silently choosing or merging them?

Permission compliance

Did retrieval exclude information the current user was not allowed to access?

Empty-result safety

What happens when suitable evidence does not exist?

Then connect those measures to the end-to-end agent evaluation.

Retrieval quality is not separate from agent quality.

It is one of its foundations.

The bigger lesson

AI agent testing becomes much more useful when we stop asking only:

Did the agent produce the expected answer?

and start asking:

What had to be true for that answer to be trustworthy?

For a retrieval-based answer, that means verifying the evidence chain:

Right query
→ Right source
→ Right version
→ Required passages
→ Correct permissions
→ Correct interpretation
→ Supported answer

A failure anywhere in that chain can produce a wrong response.

The uncomfortable part is that the response may still sound excellent.

That is why plausible retrieval failures deserve dedicated tests.

A practical test you can add today

Choose one important knowledge-based capability.

Take ten realistic user questions and, for each one, define:

1. Which evidence must be retrieved?
2. Which source version should win?
3. Which passages would be misleading?
4. Which permissions apply?
5. What should happen if required evidence is missing?

Run the retrieval without generating an answer.

Inspect the results.

You may discover problems that your prompt evaluations have never shown you.

FAQ

Should retrieval be tested separately from the final AI answer?

Yes. Testing only the final response cannot show whether the correct answer came from reliable retrieval or from a lucky model inference. Retrieval should have its own fixtures, expected sources, ranking requirements, freshness checks, and permission tests.

What is a retrieval fixture?

A retrieval fixture defines a query together with the evidence that should be returned, evidence that should not be treated as authoritative, and any ranking, version, or permission requirements.

Can a grounded AI answer still be wrong?

Yes. Grounding verifies that the answer follows the supplied evidence. If the evidence is outdated, incomplete, or inappropriate for the user, a fully grounded response can still be wrong.

What should happen when retrieval is incomplete?

For claims that require specific evidence, the agent should avoid a definitive answer. It should retrieve additional evidence, ask for necessary information, state the limitation, or escalate according to the workflow.

Should retrieval tests run when documents change?

Yes. Retrieval behaviour can change when documents, metadata, chunking, embeddings, ranking, permissions, indexes, or query transformations change—even when the prompt and application code remain untouched.

Testing the whole agent system

Retrieval is only one of the quiet failure surfaces in a production AI agent.

Memory, tools, permissions, state, evaluation systems, multi-agent coordination, and external APIs can all produce the same pattern:

Everything appears technically successful, but the final behaviour is wrong.

That is the problem I explore in The AI Agent Test Manual: how to move beyond prompt testing and build an evaluation strategy for the complete agent system.

The book covers retrieval testing, realistic regression suites, metrics, human evaluation, LLM-as-a-judge, automated evaluation pipelines, memory and tool testing, multi-agent systems, observability, incident response, and trustworthy agent design.

The AI Agent Test Manual is available here:
https://amzn.eu/d/0blIkveq

Because the agent sounding confident is not the hard problem.

The hard problem is knowing when that confidence deserves to be trusted.