Category: AI Testing

  • From 94% Pass Rate to Production Confidence: A Real Agent Evaluation Walkthrough

    From 94% Pass Rate to Production Confidence: A Real Agent Evaluation Walkthrough

    Your AI agent passes 94% of its tests. Is it ready for production? This practical evaluation walkthrough shows why the answer depends on the failures, not the headline score—and how severity gates, repeated runs, retrieval tests, deterministic checks, LLM judges, human review, latency, and cost change the decision.

  • When Three AI Agents Agree—and Are Still Wrong

    When Three AI Agents Agree—and Are Still Wrong

    Three AI agents reach the same conclusion. That sounds reassuring—until you discover that all three used the same bad evidence. This practical multi-agent testing case study shows why consensus is not truth, how correlated failures spread, and how provenance, conflict tests, deterministic checks, and termination rules improve reliability.

  • The Tool Timed Out. Did the Agent Just Do It Twice?

    The Tool Timed Out. Did the Agent Just Do It Twice?

    A tool timeout does not prove that a write failed. The action may already have happened. This practical AI agent testing example shows how naive retries create duplicate side effects, and how idempotency keys, reconciliation, explicit uncertain states, deterministic assertions, and failure injection make state-changing agent tools safer.

  • One Prompt Change, Twelve New Failures

    One Prompt Change, Twelve New Failures

    A cleaner AI prompt improved latency, cost, and overall pass rate—while quietly creating twelve new failures. This practical case study shows how prompt regressions hide behind aggregate metrics, how to build behaviour-based regression tests, and how before-and-after evaluation results can prevent a seemingly better agent from reaching production.

  • The Agent Was Confident. The Data Was Wrong.

    The Agent Was Confident. The Data Was Wrong.

    A retrieval call can succeed and still give an AI agent exactly the wrong evidence. This practical walkthrough shows how to test retrieval as its own quality surface, covering stale documents, missing exceptions, conflicting sources, query variations, evidence gates, and measurable before-and-after evaluation results.

WordPress Cookie Notice by Real Cookie Banner