From 94% Pass Rate to Production Confidence: A Real Agent Evaluation Walkthrough

The dashboard said:

94% passed

That sounds good.

Good enough to ship?

Maybe.

Suppose an AI customer-service agent passes 94 of 100 evaluation cases.

Six fail.

Before making a release decision, you need to know something much more important than the percentage:

Which six?

If the failures are awkward wording, slightly excessive verbosity, or unnecessary clarification, 94% may represent a strong candidate.

If the failures include:

• One cross-customer disclosure
• Two unsupported financial claims
• One duplicated write action
• One missed escalation
• One incorrect policy status

then 94% is not reassuring.

It is a warning disguised as a score.

This article walks through a realistic evaluation example and shows how a seemingly strong 94% pass rate becomes a much more useful production decision once we add severity, repeated runs, retrieval tests, deterministic assertions, LLM judges, human review, latency, cost, and release gates.

The numbers are illustrative. The evaluation pattern is the important part.

The agent

Imagine an insurance customer assistant that can:

• Answer policy questions
• Explain claim status
• Retrieve billing information
• Check payments
• Create callback requests
• Escalate unsupported cases

The team has built 100 evaluation cases.

They represent common customer tasks plus a smaller set of difficult situations.

The first evaluation is simple.

For each case:

1. Run the agent.
2. Compare the response with expected behaviour.
3. Mark pass or fail.

Result:

Passed: 94
Failed: 6

Pass rate: 94%

The dashboard is green.

The temptation is obvious:

Ninety-four per cent is pretty good. Let’s release.

Not yet.

Start by opening the six failures

The first improvement costs nothing.

Stop looking at the aggregate score and inspect the failures.

Suppose they are:

FAIL-01
Agent response was too verbose.

FAIL-02
Agent asked for clarification unnecessarily.

FAIL-03
Agent claimed a claim was approved although status was under_review.

FAIL-04
Agent retried a timed-out callback write and created two requests.

FAIL-05
Agent answered from an expired policy document.

FAIL-06
Agent failed to escalate when required evidence was missing.

Now 94% means something different.

Two failures are mostly quality issues.

Four affect correctness or external behaviour.

The percentage did not change.

Your release decision should.

Not all failures are equal

The simplest improvement is to classify failures by severity.

For example:

Critical

A failure that can create severe safety, privacy, security, or financial harm.

Examples:

• Cross-customer disclosure
• Unauthorised state change
• Duplicate financial action

High

Serious incorrect behaviour affecting a user or business process.

Examples:

• Unsupported customer-specific decision
• Incorrect claim status
• Missed required escalation
• Wrong policy version

Medium

A meaningful quality problem with limited harm.

Examples:

• Incomplete answer
• Unnecessary escalation
• Poor explanation

Low

Minor usability or style issue.

Examples:

• Verbosity
• Formatting
• Awkward wording

Our six failures might therefore become:

Critical: 1
High:     3
Medium:   1
Low:      1

That is a very different summary from:

94% passed

Add a hard release gate

An aggregate target is useful.

It should not be the only release rule.

Suppose the team defines:

release_gate:
  overall_pass_rate_min: 0.93
  critical_failures: 0
  high_failure_rate_max: 0.02

The candidate has:

Overall pass rate: 94%    PASS
Critical failures: 1      FAIL
High failure rate: 3%     FAIL

Release decision:

BLOCK

The system passed the headline metric and still failed the release gate.

That is exactly what the gate is for.

Then ask whether one run is enough

Agent behaviour is often nondeterministic.

A test that passes once may fail on the next run.

Take this case:

Has my claim been approved?

Tool result:

status = under_review

The expected behaviour is straightforward:

• Preserve under_review
• Do not claim approval
• Do not claim rejection
• Do not invent a completion date

One run produces:

Your claim is still under review.

Pass.

Are we finished?

Run it ten times.

Run 1   PASS
Run 2   PASS
Run 3   PASS
Run 4   FAIL - "should be approved soon"
Run 5   PASS
Run 6   PASS
Run 7   PASS
Run 8   FAIL - "approval is being finalized"
Run 9   PASS
Run 10  PASS

Now the behaviour is:

8 / 10 safe
2 / 10 unsupported claims

The agent can answer correctly.

That does not mean it does so reliably.

Reliability changes the question

For deterministic software, we often ask:

Does this test pass?

For an important agent behaviour, we may need to ask:

How often does this test pass?

Suppose ten high-risk cases are each repeated ten times.

That gives 100 executions.

Results:

92 safe
8 unsafe

The original case-level evaluation might have counted all ten cases as passing because each succeeded at least once.

Repeated execution reveals an 8% unsafe-behaviour rate.

For critical behaviours, that difference matters.

Separate generation from retrieval

One failed case involved the wrong policy document.

The final response was:

Your garden shed is covered for storm damage.

The answer was grounded.

The retrieved document supported it.

Unfortunately, retrieval had selected an expired policy version.

A response evaluator alone cannot diagnose this.

So we add a retrieval fixture:

query: "Is my freestanding garden shed covered for storm damage?"

required_evidence:
  - current detached-structure rule
  - permanent-anchoring condition

forbidden:
  - expired policy version
  - training material as authority

requirements:
  current_version_first: true

Now retrieval becomes its own test surface.

Across 20 knowledge-intensive cases:

Current-version accuracy:      85%
Required-evidence retrieval:   80%
Conflict handling:             75%

The final-answer score had hidden a weak evidence layer.

Add deterministic assertions

Not every evaluation needs an LLM judge.

Some properties are better expressed as code.

Suppose the claim tool returns:

{
  "status": "under_review"
}

and the structured output contains:

{
  "reportedStatus": "approved"
}

That should fail deterministically.

Similarly:

callback transaction count <= 1

should not depend on an LLM judge.

Neither should:

authorization = allowed

or:

required confirmation = present

or:

effective customer ID = authenticated customer ID

Use deterministic assertions for invariants.

They are cheap, fast, reproducible, and difficult to argue with.

Use LLM judges for semantic quality

Other behaviours are harder to express as simple rules.

For example:

• Did the explanation answer the user’s actual question?
• Did it communicate uncertainty clearly?
• Was the response grounded in the evidence?
• Did it omit an important condition?
• Was the escalation appropriate?
• Was the answer unnecessarily confusing?

These are useful jobs for an LLM judge.

A grounding rubric might ask the judge to classify:

SUPPORTED
PARTIALLY_SUPPORTED
UNSUPPORTED
INSUFFICIENT_EVIDENCE

But the judge also needs testing.

It is another model component, not an oracle.

Calibrate the judge with humans

Take 50 representative responses.

Have qualified human reviewers label them independently.

Then compare the judge.

Suppose:

Human / judge agreement: 88%

Critical unsafe cases:
Human detected: 10
Judge detected: 8

The judge misses two critical cases.

That may be unacceptable for a hard safety gate.

After improving the rubric and evidence package:

Human / judge agreement: 96%

Critical unsafe cases:
Human detected: 10
Judge detected: 10

Now the judge is more useful.

The important lesson is not:

Use an LLM judge.

It is:

Evaluate the evaluator.

Human review still matters

Some cases are too important, ambiguous, or novel to leave entirely to automated evaluation.

Human reviewers can inspect:

• High-risk failures
• New failure categories
• Judge disagreements
• Borderline cases
• Production complaints
• Candidate-versus-baseline changes

Suppose automated evaluation reports:

Case A: PASS
Case B: PASS
Case C: REVIEW_REQUIRED
Case D: FAIL

Human review may reveal that Case A is technically grounded but misleading.

That becomes:

Case A: FAIL - omitted critical exception

The test suite improves.

The judge can be recalibrated.

The failure becomes a regression case.

Human evaluation is not separate from automation.

It helps improve it.

Now add latency and cost

Quality is not the only production requirement.

Suppose the candidate improves grounding by performing additional retrieval and review steps.

Before:

p95 latency:        5.2 s
Cost per run:       €0.021
Quality pass rate:  94%

After:

p95 latency:        8.9 s
Cost per run:       €0.057
Quality pass rate:  98%

Is the new version better?

Probably in quality.

Maybe not operationally.

There is no universal answer.

The evaluation strategy needs explicit targets.

For example:

quality:
  pass_rate_min: 0.97
  critical_failures: 0

performance:
  p95_latency_max_seconds: 7

cost:
  average_run_cost_max_eur: 0.04

Now the trade-off becomes visible.

The candidate improves quality but fails latency and cost requirements.

That gives the team something concrete to optimise.

Measure task completion too

An agent may produce an excellent answer while failing the user’s task.

Imagine:

Please explain the payment difference and create a review request.

The agent gives a perfect explanation.

It never creates the request.

A response-quality judge might score it highly.

Task completion should fail.

For compound tasks, define required outcomes explicitly:

required_outcomes:
  - explain_payment_difference
  - create_review_request

pass:
  all_required_outcomes_completed: true

This catches partial success hidden behind fluent language.

Build a layered evaluation

We now have several layers.

Layer 1: deterministic checks

Verify:

• Schemas
• Permissions
• Entity IDs
• Tool arguments
• State transitions
• Transaction counts
• Required confirmations

Layer 2: retrieval evaluation

Verify:

• Relevant evidence
• Source authority
• Freshness
• Required exceptions
• Permissions
• Conflict handling

Layer 3: semantic evaluation

Use LLM judges for:

• Grounding
• Completeness
• Relevance
• Clarity
• Appropriate uncertainty
• Escalation quality

Layer 4: repeated execution

Measure:

• Stability
• Failure probability
• Tool-selection variance
• Loop occurrence

Layer 5: human review

Use for:

• High-risk cases
• Judge calibration
• Novel failures
• Borderline results

Layer 6: operational metrics

Track:

• Latency
• Cost
• Tool reliability
• Completion time

Layer 7: business outcome

Measure:

• Task completion
• Escalation quality
• Repeat contact
• User correction
• Human workload

Now “94% passed” has become an evaluation system.

The first before-and-after comparison

Let us return to the original candidate.

Before deeper evaluation

100 cases

Overall pass rate: 94%

Release conclusion:

Looks promising.

After inspecting severity

Overall pass rate:     94%
Critical failures:      1
High failures:          3

Release conclusion:

Block.

After repeated runs

Critical repeated cases:
Safe execution rate:   92%

Release conclusion:

Behaviour is too unstable.

After retrieval testing

Current-version retrieval:    85%
Required-evidence retrieval:   80%

Release conclusion:

Evidence pipeline needs work.

After judge calibration

Human/judge agreement: 88%
Critical judge recall:  80%

Release conclusion:

Judge is not ready to act as a hard gate.

This looks worse than where we started.

That is good.

We are replacing false confidence with information.

Fix the system, not the score

The team makes several changes:

• Adds current-version retrieval filters
• Introduces required-evidence gates
• Adds deterministic status checks
• Adds idempotency to callback creation
• Restores explicit uncertainty behaviour
• Improves escalation logic
• Recalibrates the grounding judge
• Adds production failures to the regression suite

Then the evaluation runs again.

After: the candidate earns confidence

The revised candidate produces:

Overall pass rate:             99%
Critical failures:              0
High failure rate:              1%

Critical repeated-run safety: 100%

Current-version retrieval:    100%
Required-evidence retrieval:   98%
Conflict handling:             97%

Human/judge agreement:         96%
Critical judge recall:        100%

p95 latency:                  6.3 s
Average cost per run:        €0.034

Duplicate writes:                0
Unauthorised actions:            0

Now the team has a much stronger basis for release.

Not because 99% is a magical number.

Because the evidence covers the failure modes that matter.

Before and after

The most important change is not:

94% → 99%

It is:

Before:
"We ran 100 tests and 94 passed."

After:
"We know which behaviours are critical.
Critical failures are zero.
Repeated high-risk cases are stable.
Retrieval returns the required current evidence.
Write actions are protected.
The judge is calibrated against humans.
Latency and cost meet targets.
The complete candidate passes the release gate."

Those are completely different claims.

The second one is much closer to production confidence.

Build a release scorecard

A useful release report could look like this:

QUALITY
Overall pass rate             PASS
Critical failures             PASS
High-severity failure rate    PASS

EVIDENCE
Required retrieval            PASS
Freshness                     PASS
Conflict handling             PASS

SAFETY
Authorization                 PASS
Confirmation                  PASS
Duplicate-write protection    PASS

STABILITY
Repeated critical cases       PASS
Loop limits                   PASS

EVALUATION
Judge calibration             PASS
Human review                  PASS

OPERATIONS
p95 latency                   PASS
Cost                          PASS
Trace completeness            PASS

Now a release decision is explainable.

That matters to engineers.

It also matters to product owners, security teams, operations, auditors, and anyone asked:

Why did you believe this agent was ready?

Avoid one giant score

You can combine metrics into a single score.

Sometimes that is useful for dashboards.

But be careful.

Imagine:

Quality:      99
Latency:      95
Cost:         98
Safety:       40

A weighted average could still look respectable.

That does not make the system safe.

Some requirements are not trade-offs.

They are gates.

An unauthorised financial action should not be compensated for by excellent response quality.

Use scores for optimisation.

Use hard gates for unacceptable failure.

Production confidence is never permanent

Even after release, the evidence can decay.

The system may change because of:

• Model updates
• Prompt changes
• New documents
• Retrieval changes
• Tool versions
• User behaviour
• External APIs
• New attacks
• Memory growth

The evaluation therefore continues in production.

Use:

• Production sampling
• Behavioural metrics
• Trace review
• User feedback
• Incident replay
• Scheduled evaluations
• Drift detection

Production confidence is maintained, not achieved once.

What should a team evaluate first?

You do not need a perfect evaluation platform before testing your first agent.

Start with the risks.

For one important capability, define:

1. What does success mean?
2. What failures are unacceptable?
3. Which evidence is required?
4. Which tools may be used?
5. Which actions require permission or confirmation?
6. What happens when dependencies fail?
7. Which metrics matter operationally?

Then build tests around those answers.

A smaller risk-based suite is usually more valuable than 1,000 generic prompts.

FAQ

Is a 90% or 95% AI agent pass rate good enough for production?

There is no universal threshold. A pass rate is only meaningful when you know which tests failed, how severe those failures are, how representative the dataset is, and whether critical behaviours have hard release gates.

Why should AI agent tests be repeated?

Agent behaviour can vary between runs. Repeating important cases reveals instability and failure probabilities that a single successful execution can hide.

Should LLM judges replace human evaluation?

No. LLM judges can scale semantic evaluation, but they should be calibrated against human reviewers and monitored for disagreement, drift, and critical misses.

What should be tested besides the final response?

Test retrieval, memory, permissions, tool use, arguments, state transitions, retries, side effects, task completion, latency, cost, and traceability. The final response is only one output of the agent system.

What is an AI agent release gate?

A release gate is an explicit rule that determines whether a candidate can progress. Examples include zero critical failures, a minimum quality threshold, maximum latency, required retrieval accuracy, and no unauthorised actions.

Why are hard gates better than one overall score?

Some failures should not be traded against other improvements. A privacy breach or unauthorised transaction should block a release even when average quality, latency, and cost are excellent.

The number is the beginning, not the answer

A pass rate is useful.

It tells you something.

But it does not tell you enough to decide whether an AI agent deserves production responsibility.

When someone says:

The agent passes 94% of our tests.

the next questions should be:

Which 6% failed?

How serious are those failures?

Were the important cases repeated?

Did retrieval return the right evidence?

Were side effects checked?

Was the judge calibrated?

Did humans review the risky cases?

Does the system meet latency and cost targets?

What blocks the release automatically?

Those questions turn evaluation from a benchmark into engineering.

That is the approach behind The AI Agent Test Manual: practical evaluation for the complete agent system, from test-case design and metrics to retrieval, tools, LLM judges, automated pipelines, observability, incident response, and trustworthy production operation.

The AI Agent Test Manual:
https://amzn.eu/d/0blIkveq

A 94% pass rate might be excellent.

It might be terrible.

The percentage cannot tell you.

The failures can.