Tag: AI Reliability
From 94% Pass Rate to Production Confidence: A Real Agent Evaluation Walkthrough
Your AI agent passes 94% of its tests. Is it ready for production? This practical evaluation walkthrough shows why the answer depends on the failures, not the headline score—and how severity gates, repeated runs, retrieval tests, deterministic checks, LLM judges, human review, latency, and cost change the decision.
The Tool Timed Out. Did the Agent Just Do It Twice?
A tool timeout does not prove that a write failed. The action may already have happened. This practical AI agent testing example shows how naive retries create duplicate side effects, and how idempotency keys, reconciliation, explicit uncertain states, deterministic assertions, and failure injection make state-changing agent tools safer.
One Prompt Change, Twelve New Failures
A cleaner AI prompt improved latency, cost, and overall pass rate—while quietly creating twelve new failures. This practical case study shows how prompt regressions hide behind aggregate metrics, how to build behaviour-based regression tests, and how before-and-after evaluation results can prevent a seemingly better agent from reaching production.
The Agent Was Confident. The Data Was Wrong.
A retrieval call can succeed and still give an AI agent exactly the wrong evidence. This practical walkthrough shows how to test retrieval as its own quality surface, covering stale documents, missing exceptions, conflicting sources, query variations, evidence gates, and measurable before-and-after evaluation results.
To: All Staff – Subject: The LLM Outage of 2028
When the world’s LLMs crashed, so did coffee orders, standups, and even our motivational quotes. This fictional memo from the future reveals what happens when we trust AI just a bit too much—and forget how to type for ourselves.
