One Prompt Change, Twelve New Failures

The pull request looked harmless.

The old prompt had become repetitive. Several rules appeared twice. Some instructions had grown organically over months of fixes. Responses were longer than they needed to be.

So the team cleaned it up.

The new prompt was shorter, clearer, and easier to maintain.

Initial results looked excellent.

Response time improved.

Token usage dropped.

Answers became shorter.

The agent sounded more decisive.

Then the regression suite ran.

Twelve tests that had passed the day before now failed.

The prompt had become better.

The agent had become worse.

That is one of the most useful lessons in AI agent testing:

A prompt change is a software change, even when it looks like editing.

And like any software change, it can break behaviour far away from the sentence you changed.

The change

Imagine a customer-service agent that handles insurance claims.

Its original instructions include several overlapping rules:

Answer the customer's question clearly.

When the available evidence is incomplete,
say what is known and what remains uncertain.

Do not describe a claim as approved, rejected,
or paid unless that status is verified.

If required information is unavailable,
explain the limitation.

Escalate when a definitive answer cannot be supported.

Keep answers concise where possible.

After months of additions, similar rules appear in several places.

Someone reasonably decides to simplify them.

The revised section becomes:

Answer directly and concisely.

Use the available customer and claim information.

Do not invent information.

Escalate when necessary.

Nothing here looks obviously dangerous.

The new instructions still say not to invent information.

They still mention escalation.

They are shorter and easier to understand.

A reviewer might approve the change immediately.

The first evaluation looks better

The team runs 50 ordinary test conversations.

The results are encouraging:

Before                    After

Pass rate       92%       94%
Avg. latency    5.8 s     5.1 s
Avg. tokens     1,420     1,180
Avg. length     168 words 132 words

The new prompt appears to be an improvement.

If these were the only metrics, the release would probably ship.

But the 50 conversations mostly represent common successful paths.

They ask things such as:

What is the status of my claim?

When will my payment arrive?

Which documents are still missing?

When the required data exists, both prompt versions perform well.

The differences appear at the boundaries.

Then the regression suite runs

The regression suite contains cases collected from earlier failures, production incidents, edge cases, and deliberately difficult scenarios.

Twelve cases fail after the prompt change.

Not twelve variations of one bug.

Twelve failures across four different behaviours.

Failure 1: uncertainty disappeared

Tool result:

claim_status = under_review
decision = unknown

Old response:

Your claim is still under review. I cannot confirm a decision yet.

New response:

Your claim is currently being reviewed and should be completed soon.

The second answer sounds better.

It also introduces a claim that the evidence does not support.

There is no information about when the review will finish.

The new prompt encouraged directness without preserving the previous uncertainty requirement.

Failure 2: partial results became complete results

Suppose an address update spans three systems:

Customer profile: updated
Home policy: updated
Travel policy: processing

Old response:

Your customer profile and home policy have been updated. The travel-policy update is still processing.

New response:

Your address has been updated successfully.

Nothing was invented from thin air.

The operation had succeeded partially.

The shortened answer simply collapsed a composite state into one convenient status.

That is still wrong.

Failure 3: missing evidence no longer triggered escalation

A customer asks:

Why was €260 deducted from my reimbursement?

The payment system returns:

requested_amount = 900
approved_amount = 640

But the policy retrieval service fails to return the rule explaining the difference.

Old behaviour:

I can confirm that €640 was approved, but I do not have enough information to explain the €260 difference. I can have this reviewed.

New behaviour:

Your reimbursement was reduced because part of the claim was not covered.

Plausible.

Probably even likely.

Unsupported.

Failure 4: clarification became assumption

A customer has two active claims and asks:

Has the payment gone out yet?

The original agent asks which claim the customer means.

The revised agent selects the most recently viewed claim and answers directly.

Again, the new response is faster and more convenient.

It is also attached to an assumption the user never confirmed.

Why did a small prompt change cause this?

The team had not removed a single obvious safety instruction.

Instead, it removed several behavioural constraints that looked redundant when read individually.

That distinction matters.

Prompts are not traditional programs. Their behaviour emerges from the interaction between instructions, examples, context, tools, model behaviour, and the user’s request.

Removing one sentence may change the weight of another.

Changing:

Keep answers concise where possible.

into:

Answer directly and concisely.

looks minor.

But it can alter how the model resolves a tension between completeness and brevity.

If uncertainty, evidence requirements, and escalation are not independently tested, the regression is easy to miss.

Aggregate pass rate hid the problem

The headline score improved from 92 to 94 per cent.

That sounds like evidence for release.

But the score treats every test as equally important.

Imagine the four remaining failures in the old version were mostly formatting and verbosity issues.

The six failures in the new version include unsupported customer-specific claims.

The new agent can have a higher pass rate while being less trustworthy.

This is why I rarely want a single quality score to decide whether an agent ships.

The important question is not only:

How many tests failed?

It is also:

Which behaviours failed?

Classify regression tests by severity

Suppose the regression suite contains 120 tests.

Before the prompt change:

Critical failures:     0
High failures:         1
Medium failures:       4
Low failures:          6

After the prompt change:

Critical failures:     2
High failures:         5
Medium failures:       3
Low failures:          2

The total failure count has barely changed.

The risk profile has changed dramatically.

That should block the release.

A useful release gate could be:

release_gate:
  critical_failures: 0
  high_failure_rate_max: 0.02
  overall_pass_rate_min: 0.93
  latency_regression_max: 0.10
  cost_regression_max: 0.15

Now the system cannot compensate for a critical behavioural regression by becoming cheaper or more concise elsewhere.

What should be in a prompt regression suite?

A good regression suite should preserve behaviour that matters, not wording.

For a customer-specific agent, I would include cases covering uncertainty, incomplete evidence, conflicting evidence, multiple entities, permissions, tool failure, partial completion, confirmation, escalation, unknown statuses, repeated actions, and previous incidents.

The expected answer does not always need to be exact text.

For example, this is too brittle:

Expected response:
"I cannot confirm whether your claim was approved."

A more durable test defines behaviour:

required:
  - state that approval is not verified
  - preserve under_review status

forbidden:
  - claim approved
  - claim rejected
  - invent completion time

acceptable_actions:
  - ask for more information
  - offer approved escalation

Now the agent may phrase the answer naturally while the important behaviour remains testable.

Test claims, not just responses

One particularly useful technique is to extract the material claims from a response.

Suppose the agent says:

Your claim is still being reviewed and should be completed within the next few days.

That sentence contains at least two claims:

Claim 1:
The claim is under review.

Claim 2:
The review should complete within a few days.

The first may be supported.

The second may not be.

A response-level judge might score the answer as mostly correct.

A claim-level evidence check exposes the unsupported addition.

For high-risk answers, that distinction is valuable.

Use deterministic checks where possible

Not every regression needs another LLM call.

Suppose a claim-status tool returns:

{
  "status": "under_review"
}

and the final structured response contains:

{
  "reportedStatus": "rejected"
}

You do not need an LLM judge to recognise the contradiction.

A deterministic assertion can fail immediately.

Likewise, a state-changing action can be checked for:

confirmation_received = true
authorization = allowed
transaction_count = 1

Use models for semantic evaluation where necessary.

Use ordinary code for deterministic invariants.

Add repeated runs for unstable cases

A prompt regression may not fail every time.

Suppose one test behaves like this over ten runs:

Old prompt:

10 / 10 safe responses
0 unsupported claims

After the change:

New prompt:

7 / 10 safe responses
3 unsupported claims

A single evaluation run may still pass.

That is why important nondeterministic behaviours need repeated execution.

For critical cases, I care less about whether the agent can produce the correct response than whether it does so reliably.

The before-and-after result

After analysing the twelve failures, the team does not simply restore every deleted sentence.

Instead, it separates important behavioural requirements from stylistic guidance.

The revised prompt keeps the concise style but restores explicit rules around evidence, uncertainty, status claims, and escalation.

The team also adds deterministic checks around customer-specific status claims.

Then the complete suite runs again.

                         Original   Shortened   Revised

Overall pass rate           92%        94%        98%
Critical failures             0          2          0
High failures                 1          5          0
Unsupported claims            2          9          1
Avg. latency                5.8 s      5.1 s      5.2 s
Avg. tokens                1,420      1,180      1,210

The revised version keeps most of the efficiency gain.

More importantly, it recovers the behaviour that mattered.

This is a much stronger improvement than simply returning to the old prompt.

The incident produced better instructions and better controls.

The bigger lesson: do not test prompt text

Test behaviour.

A prompt may change because you:

• Shorten instructions
• Add examples
• Change formatting
• Change priority
• Update tool descriptions
• Move instructions
• Switch models
• Reduce context
• Add a new capability

The correct regression question is not:

Does the new prompt still contain the old safety sentence?

It is:

Does the system still exhibit the behaviour that sentence was meant to create?

That is the level at which regression testing becomes useful.

A simple prompt-change workflow

For every meaningful prompt change, identify which behaviours could plausibly be affected. Run targeted regression cases first, then the critical suite, then compare the candidate with the current baseline.

Look at more than total pass rate. Compare failure categories, unsupported claims, tool behaviour, escalation, latency, and cost.

A candidate should be able to answer two questions before release:

What became better?

and:

What became worse?

If you cannot answer both, you do not yet understand the change.

Keep production failures forever

The most valuable regression cases are often not synthetic.

They are the cases your system actually got wrong.

Every meaningful incident should become a test.

A customer-specific claim was reported as rejected while the tool said under_review?

Keep it.

A timeout caused the agent to retry a write?

Keep it.

A stale memory value overrode current system data?

Keep it.

A retrieval result omitted the one paragraph containing the exception?

Keep it.

Over time, your regression suite becomes a record of what your system has learned not to do.

That is far more valuable than a collection of generic “good chatbot” examples.

FAQ

Should every prompt edit run the full evaluation suite?

Not necessarily. Test selection should depend on what changed and which capabilities it can affect. Small wording changes may run targeted evaluations plus the critical regression suite. High-impact changes should trigger broader testing.

Why can a higher pass rate still mean a worse agent?

Because aggregate pass rates hide severity. Ten harmless formatting failures may matter less than one unauthorised action or unsupported financial claim. Release decisions should consider failure categories and hard safety gates.

Should regression tests expect exact wording?

Usually not. Exact strings are brittle for generative systems. Test required facts, forbidden claims, evidence use, tool behaviour, state transitions, and other observable properties instead.

How many times should an AI agent regression test run?

It depends on risk and variability. Deterministic behaviours may need one run. Important nondeterministic behaviours should be repeated enough to detect instability that a single successful execution could hide.

Can deterministic tests be used with LLM agents?

Absolutely. Authentication, permissions, tool arguments, state transitions, transaction counts, schema validation, and many evidence relationships can be checked deterministically. LLM judges should complement these controls rather than replace them.

Test the change, not the intention

The engineer who shortened the prompt did not make an unreasonable change.

The revised instructions were cleaner.

The common-path evaluation improved.

Latency dropped.

Cost dropped.

Nothing about the change looked reckless.

That is exactly why regression suites matter.

You do not build them only for changes you expect to be dangerous.

You build them because changes can have consequences you did not predict.

This is one of the central ideas behind The AI Agent Test Manual: AI agents need the same discipline we expect from other production systems, adapted for nondeterministic behaviour.

The book covers evaluation strategy, regression suites, metrics, human evaluation, LLM judges, automated release gates, retrieval and memory testing, tools and APIs, multi-agent systems, observability, incident response, and trustworthy agent design.

The AI Agent Test Manual:
https://amzn.eu/d/0blIkveq

A cleaner prompt is useful.

A faster agent is useful.

A cheaper agent is useful.

But none of those improvements matters if the system quietly forgets how to say:

I do not have enough evidence to know.