A Failed Call Is a Late Test

How Long Does a Production Loop Take to Find a Rare Failure?

A production loop will find the failure modes in order of frequency rather than order of cost. If a given failure mode occurs on a fraction p of calls, then the probability that it will have occurred at least once after n calls is 1 − (1 − p)^n and the expected wait for the first occurrence is 1/p calls.

Take a modest deployment handling 1,000 calls a day, about 30,000 a month:

The common failures, a slow turn or a misheard digit, surface on day one and get fixed. The rare ones arrive months later, one customer at a time.

In a contact centre, the rare paths are often the regulated ones: a bereavement policy, a cancellation right, a mandatory disclosure, a fraud escalation. Low traffic, high consequence. More volume does not rescue this: at 100,000 calls a day the 1-in-100,000 path shows up daily, and so does a new customer on the wrong end of it.

There's a second trap here. Simulating calls from the production distribution doesn't solve it, as it suffers from the same blind spot: to see a 1-in-10,000 behaviour via sampling, you still need approximately 10,000 samples.

What solves it is targeting. If the policy document states that bereavement fares exist, then a scenario that inquires about bereavement fares will hit that path with probability very close to one. Specification turns the rare-in-traffic into the certain-in-test.

Why Do Production Loops Miss the Most Expensive Failures?

A production loop doesn't ingest failures. It ingests the failures that have been flagged.

The worst conversational failures are the ones that no one actually flags at the time. A caller who just hangs up happy with a wrong answer is a call that has no escalation, no angry sentiment and a fairly good CSAT.

The most well-documented example is a chat rather than a call. On 11th November 2022 (the day his grandmother passed away), Jake Moffatt asked the Air Canada website chatbot about bereavement fares and was told that he could claim the fare after he'd travelled. The actual Air Canada policy didn't allow for retroactive claims.

On the basis of tone, sentiment and resolution, the chat would probably have been a success. The only way the failure became apparent was when he went to actually apply for the refund and was turned down.

On 14 February 2024 (some 15 months after the chat), the BC Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation in Moffatt v. Air Canada, 2024 BCCRT 149. The Tribunal rejected the argument that the chatbot was responsible for its own statements.

The award was C$812.02. The precedent cost was enormous.

Two lessons. First, the signal for a confident wrong answer isn't in the tone of the call at all; it's in the gap between what the agent says and what the policy says. Second, that gap can only be verified against an independent reference point, the policy document, which is there prior to the first call.

A production monitor who is grounded in the policy can catch the second customer. Only a pre-launch test grounded in the policy will catch the first.

Can You Replay a Failed Call as a Regression Test?

A recorded call is just one path through the conversation, and that path is invalid at the first point where the new agent version of the agent gives a different answer. If you play back the caller's original lines after that point, you're talking past an agent who has already moved on. The replay is then testing the script rather than the agent.

A failed call can't be promoted to a test as it is. It has to be broken down into the constituent parts that would survive a different agent response:

• Intent. What the caller wanted to achieve, irrespective of the way it was worded.
• Caller conditions. Accent, pace, background noise, interruptions, the digit they mumbled.
• Constraints. The facts which bound a correct answer: the policy clause, the account state and the regulatory script.
• Success criteria. What must be true at the end of the call, in such a way that a verifier can say yes, no, or unable to verify, supported by the quote.

Only the fourth section constitutes a test. Without written criteria, "did it pass" is re-evaluated on every run, and two runs of the same call can in fact yield different results. The issue isn't who is doing the grading (a human or a model); it's the grading without a fixed rubric and without a quoted line to point to.

If you have a human evaluator who is held to the written criteria and to the evidence and who is independent of the agent being evaluated, then they will return the same verdict on the same transcript. That's what makes a pass mean anything.

Notice what this dissection is: it's a specification. Turning a failed call into a test is the act of writing the specification that should have been in place prior to the call.

A team that does this will end up with a specification. A team that starts from a specification will have most of these tests in place before launch.

How Many Reruns Prove an Agent Fix? (pass^k and the Rule of Three)

The standard way to close a production failure is to fix the prompt, re-run the failed scenario, see it pass, and then ship. For a stochastic agent, a single pass is very weak evidence. An agent that succeeds on a task 70% of the time will pass a single rerun 70% of the time regardless of whether or not the fix actually worked.

The τ-bench authors actually did build a metric for this. pass@k asks whether any of the k attempts succeeds. pass^k asks whether all k attempts succeed, which is precisely what a customer-facing agent has to do every single day.

The results below are from simulated retail and airline customer service tasks, all run in text. A voice agent would of course carry that variance along with speech recognition variance and turn-taking variance on top of it.

The agent is stochastic. The verdict must not be.

That has two practical consequences: each criterion is assessed in the same way on every run, against the same evidence. The number of runs is determined by the level of confidence you require, not by some arbitrary habit.

The stats here are very basic and quite old. If a fixed scenario goes through n times in a row with zero failures, the 95% upper bound on its true failure rate is approximately 3/n (the rule of three).

To assert that a fix has a failure rate below 5% you need about 60 clean runs. Below 1%, you need about 300. With just one clean rerun, the exact 95% upper bound on the failure rate is 95%, and it rules out absolutely nothing.

Why Do Voice Agents Regress Without a Deployment?

A production-to-test loop is normally wired to deploy (change the prompt, run the suite, gate the release). That assumes the agent only changes when you change it. Hosted models, speech vendors and voice providers all break that assumption.

Stanford and Berkeley researchers looked at the March and June 2023 versions of GPT-4 on the same prompts and found that behaviour changed dramatically across different tasks. On one task (prime number identification), accuracy dropped from 84% to 51%, and the authors attributed a lot of the drift to poorer instruction following.

Critics subsequently argued that some of that was simply a change in behaviour rather than a change in capability. For an agent built on that endpoint, the distinction doesn't matter at all. The same prompt would yield different answers, and no one had shipped anything.

A voice agent stacks a whole host of dependencies on top of each other: speech-to-text, the language model, text-to-speech, the telephony carrier and every tool API it calls. Any one of those can go under a frozen prompt.

A suite that only runs on deploy will show green until a customer reports red. It has to run on a clock as well as on a commit.

How Do You Turn Production Failures Into Regression Tests That Actually Work?

The fix isn't to stop using production as a test source. The fix is to stop treating production as the first source.

The specification is the seed for the suite prior to launch. Production recordings are used to audit the specification post-launch. Both are then assessed by the same verifier against the same criteria.

That's how TestMu AI Voice Agent Testing is built. It's a single platform that can test chat, voice, inbound and outbound phone, video and image agents.

It tests them from the outside via their endpoint or phone number with absolutely no code changes to the agent. Agents behind a firewall are accessed via a secure tunnel.

The detail that matters most is in rows 3 and 4: pre-launch calls and production recordings are scored by the same metrics. That makes them comparable.

When a production score drops on a metric the pre-launch suite rated well, the gap is not noise. It is a precise statement of what the specification failed to cover, and therefore of the next scenario to write.

The second point is where the reference comes from. A suite built purely from the agent's own production history is essentially the agent judging itself.

A suite that's seeded from the policy, the PRD and the ticket is the agent being judged against what the business decided it should do. The model never rates its own work. For the mechanics of comparing the verdicts across versions, see the guide to agent regression testing.

Which Metric Shows That Testing Is Ahead of the Customer?

If you're running an interactive agent, then the one ratio you should be tracking is the ratio of distinct failure classes that you've found prior to production. That's your pre-production discovery ratio.

It's a lower bound by definition (only counts the failures that someone has actually found). The trend in that ratio is the point really; it tells you whether your testing is getting ahead of the customers or whether it's just following them.

A team whose suite only grows out of failed calls will have that ratio hovering near zero by construction regardless of how fast their loop runs. Every class they know about will have cost at least one customer. A team that seeds from the spec and audits against production will see the ratio increase with every release and should consider every production-only failure to be a defect in the spec rather than just a defect in the agent.

The speed of the loop still matters. A quicker loop that kicks off in production is actually a quicker way to learn from customers who have already been let down. The aim is to get to a loop where production is largely confirming what the testing phase already knew.

To see the two-way loop on your own agent, start with TestMu AI Voice Agent Testing: seed scenarios from your spec, run the real calls, and score the production recordings against the same criteria.

Sources

• Yao et al., τ-bench: Benchmark for Tool-Agent-User Interaction inside Real-World Domains (2024); pass^k values via benchmarklist.com
• Chen, Zaharia and Zou, How Is ChatGPT's Behavior Changing Over Time?, Harvard Data Science Review (2024)
• Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal decision (February 2024)
• BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot, ABA Business Law Today (February 2024)
• Air Canada Held Responsible for Chatbot's Hallucinations, AI Business (February 2024)
• Inbound Phone Agent Testing With TestMu AI, TestMu AI documentation
• Discovery-rate and rule-of-three figures are computed in this piece from 1 − (1 − p)^n and the 3/n bound.