Category: AI Testing
When Three AI Agents Agree—and Are Still Wrong
Three AI agents reach the same conclusion. That sounds reassuring—until you discover that all three used the same bad evidence. This practical multi-agent testing case study shows why consensus is not truth, how correlated failures spread, and how provenance, conflict tests, deterministic checks, and termination rules improve reliability.
The Tool Timed Out. Did the Agent Just Do It Twice?
A tool timeout does not prove that a write failed. The action may already have happened. This practical AI agent testing example shows how naive retries create duplicate side effects, and how idempotency keys, reconciliation, explicit uncertain states, deterministic assertions, and failure injection make state-changing agent tools safer.
One Prompt Change, Twelve New Failures
A cleaner AI prompt improved latency, cost, and overall pass rate—while quietly creating twelve new failures. This practical case study shows how prompt regressions hide behind aggregate metrics, how to build behaviour-based regression tests, and how before-and-after evaluation results can prevent a seemingly better agent from reaching production.
The Agent Was Confident. The Data Was Wrong.
A retrieval call can succeed and still give an AI agent exactly the wrong evidence. This practical walkthrough shows how to test retrieval as its own quality surface, covering stale documents, missing exceptions, conflicting sources, query variations, evidence gates, and measurable before-and-after evaluation results.
