How to Test a Voice Agent: A Step-by-Step Test Plan

Step 1: Write the Voice Agent's Test Contract Before the First Call

Before the first call you need a written test contract and access to the places where the evidence will be. The test contract is one page that states what the agent must do, what it must never do, which details it must capture exactly and when it must hand the caller to a person.

Each line has to be checkable from outside the agent, because every test case in Step 3 is one of these lines turned into a call. The example column below is for a fictional clinic's appointment-booking agent, which the rest of this guide reuses.

The numbers in the last two rows are that fictional team's own example thresholds. They are not industry standards, and Step 5 explains why you set your own.

Write the contract lines for complex and urgent requests with the most care, because those are the requests people bring to a phone line. In Five9's 2025 Customer Experience Report, a vendor survey of 1,006 respondents in the US, UK and Canada conducted by Zogby Analytics, 56% of consumers preferred phone support for general issues and 74% for complex or urgent matters.

Then confirm you can reach the evidence:

• A test number or endpoint - separate from production, so that test bookings never reach real customers.
• Recordings and transcripts - for every test call, with timestamps.
• Read access to the system of record - the calendar, CRM or database the agent writes to.
• Test accounts - patients and appointment slots you can create and reset between runs.
• A destination phone you control - to answer the transfers in Step 6.

Exit criterion - every line of the contract names something outside the agent that can prove it: a recording, a transcript, a phone you control or a record in a system.

Step 2: Check the Agent's Logic in Text, Then Place Manual Calls

Check the logic in text first, because a text run is quick to repeat and takes the audio out of the question. Then place calls by hand, because only a call exercises hearing, timing and the phone line.

The platforms' own documentation runs in the same order: Retell AI, for example, lists "several ways to test an agent before it takes real calls, from a quick text chat to a full phone call".

In the text console, type through:

• Every task in the contract from start to finish, checking the tool calls and their arguments if the console shows them.
• Every hard rule, by trying to make the agent break it.
• One correction in the middle of a task, such as a changed date.
• One question the agent has no source for.

If the same agent also answers in a chat widget, how to test a chatbot covers that channel.

Then call the agent yourself for each task: once through the platform's web call, if it has one, and once from a mobile phone to the test number. A web call lets you hear the audio, the delay and the interruptions, and a phone call adds carrier audio, the keypad and transfers.

On each call, listen for:

• How long the greeting takes to start, and whether it starts over your "hello".
• Whether the agent answers before you finish, or leaves a long gap after you stop.
• Whether it reads back every date, time and number.
• What it does when you say nothing.
• How it pronounces names, times and reference numbers.
• How the call ends, and which side hangs up.

Log every call in one sheet with these columns: call ID and time, number or endpoint, prompt and model version, the task and what you did as the caller, the expected outcome, the actual outcome in the system of record, the first turn that went wrong with its timestamp, the recording link and a verdict.

After a failed call, change one thing and place the same call again before you touch anything else.

Exit criterion - every task in the contract has completed on a real phone call, and every failure is in the log with its recording.

Step 3: Build Voice Agent Test Cases With Pass Criteria You Can Observe

A voice agent test plan covers core tasks, changes of mind, audio conditions, interruptions and silence, telephony, hand-off triggers, tool failures, out-of-scope questions, adversarial callers and compliance. In this guide a test case is one row of the table below, and a scenario is that row written out as a call.

Write each pass criterion so that someone who cannot see inside the agent can check it. The last column names where the evidence is.

Step 6 explains rows T-01 to T-05 and H-01, and Step 7 rows S-01 and O-01. For more attacks to add beside X-01 and X-02, see prompt injection testing.

Rows R-01 to R-04 rest on the texts cited below. None of this is legal advice, and whether a rule applies to your calls is a question for counsel.

• R-01, United States - the FCC's Declaratory Ruling FCC 24-17 (February 8, 2024) confirms that the TCPA's restrictions on "artificial or prerecorded voice" cover "current AI technologies that generate human voices", so such calls "require the prior express consent of the called party" unless an emergency purpose or exemption applies.
• R-02, European Union - Article 50(1) of the AI Act, as published on the European Commission's AI Act Service Desk, has providers ensure that people are "informed that they are interacting with an AI system, unless this is obvious", and Article 50(5) sets the timing: "at the latest at the time of the first interaction or exposure". Article 113 of the same regulation gives 2 August 2026 as its general application date. The duty is addressed to providers of AI systems.
• R-03, United States - in FCC 24-84 (August 2024) the FCC proposed "requiring callers using AI-generated voice to, at the beginning of each call, clearly disclose to the called party that the call is using AI-generated technology". The FCC's regulatory agenda of August 14, 2026 lists the next action on the proposal as undetermined, so this guide treats R-03 as a policy you set yourself.
• R-04, card payments - the PCI Security Standards Council's guidance on telephone-based payment card data (version 3.0, November 2018) says "controls should be in place to ensure that SAD is either not recorded or, if it is recorded, that it is securely deleted immediately upon authorization of the transaction". SAD is sensitive authentication data, which includes the card security code. The guidance predates voice agents, and applying it to transcripts is this guide's extension.

Size the table by coverage: beyond one case per contract line, repeat every case that captures a detail under each audio condition your callers have, and add a row for every failed production call. Once you have real call logs, the voice agent testing guide shows how to build a golden call set from them.

Exit criterion - every contract line maps to at least one row, and every row's pass criterion names where its evidence is.

Step 4: Test Accents, Background Noise and Line Quality

Run the same scenario under different voices, noise levels and line quality, changing one condition at a time. Score what the speech recognizer heard separately from what the agent decided, so that each failure can be assigned to hearing or to logic.

• Voices - use the accents, ages and speaking speeds found among your own callers. Choosing the groups is covered in accent testing for voice agents.
• Noise - replay the scenario over recordings from the places your callers phone from, at more than one level. Building that set is covered in background noise testing for voice agents.
• Line quality - repeat the scenario over a narrowband phone line and a weak mobile connection. Jitter, packet loss and the MOS scale are covered in voice quality testing.
• Word error rate - a NIST slide on speech recognition metrics from 2000 defines it as the sum of "word insertions + word deletions + word substitutions" divided by the "total words in reference". Compute it on the caller side of the agent's own transcript against your script, as a measure of the recognizer.
• Captured details - compare each value the agent wrote to a system or read back, such as a date, a time or a number, with the script. This measures what reaches your systems.

To put a chosen voice or noise on a call by hand, prepare each caller turn as an audio file (a recorded speaker or a text-to-speech voice), mix the noise in beforehand, and play the files into the call from a softphone or a virtual audio device. A simulated caller with a configurable voice and background noise (Step 9) does the same without the files.

Read the transcripts before you trust either number. The Whisper paper (Radford et al., 2022) notes that word error rate "penalizes all differences between the model's output and the reference transcript including innocuous differences in transcript style".

# pink noise, fixed seed, exactly as long as the clip (86800 samples here)
ffmpeg -f lavfi -i "anoisesrc=color=pink:seed=1001:sample_rate=16000:amplitude=1:duration=6.425" -af "atrim=end_sample=86800" -c:a pcm_f32le audio/noise/david-u01.wav

# the phone step: 300 to 3400 Hz, 8 kHz, G.711 mu-law
ffmpeg -i audio/prephone/phone_snr10/david-u01.wav -af "aformat=sample_fmts=fltp,highpass=f=300,lowpass=f=3400,aresample=8000" -c:a pcm_mulaw audio/mulaw/phone_snr10/david-u01.wav

# back to 16 kHz PCM for the recognizer
ffmpeg -i audio/mulaw/phone_snr10/david-u01.wav -ar 16000 -c:a pcm_s16le audio/phone_snr10/david-u01.wav

Speech recognition under phone-line conditions
recognizer: ffmpeg version 8.0.1-full_build-www.gyan.dev, whisper filter, ggml-base.en.bin
clips: 144 = 12 sentences x 2 synthetic voices x 6 conditions

condition    clips  words    S    D    I     WER   no word errors    slots ok
wideband        24    214    0    2    0    0.9%         22 of 24    24 of 24
phone           24    214    2    2    0    1.9%         21 of 24    23 of 24
phone_snr20     24    214    0    2    0    0.9%         22 of 24    24 of 24
phone_snr10     24    214    2    0    0    0.9%         22 of 24    24 of 24
phone_snr5      24    214    7    0    0    3.3%         19 of 24    23 of 24
phone_snr0      24    214   21    3    4   13.1%         13 of 24    22 of 24

words = words in the reference sentences after normalization (a digit string counts as one word)
S, D, I = substituted, deleted and inserted words; WER = (S + D + I) / words
slots ok = clips in which every value the agent needs was heard, in spoken order

Slot misses: 4 of 144 clips

phone  zira  u10  missing: 30, 13
  said:  No, that is not right. I said the thirtieth, not the thirteenth.
  heard: No, that is not right. I said the 30s, not the 13s.
  note:  the digits are there, inside a different word (30s, 13s)

phone_snr5  david  u01  missing: 15
  said:  I would like to book an appointment for Tuesday the fifteenth at four thirty in the afternoon.
  heard: I would like to book an appointment for Tuesday at 14th at 4.30 in the afternoon.

phone_snr0  david  u01  missing: 15
  said:  I would like to book an appointment for Tuesday the fifteenth at four thirty in the afternoon.
  heard: I would like to book an appointment for Tuesday at 14 at 4.30 in the afternoon.

phone_snr0  zira  u10  missing: 30, 13
  said:  No, that is not right. I said the thirtieth, not the thirteenth.
  heard: No, that is not right. I said the third year, not the third change.

Exit criterion - for each condition your callers have, you know the share of calls with every detail captured, and you can name the condition where that share starts to drop.

Step 5: Test Interruptions, Silence and Response Time

Test turn-taking by doing on purpose what callers do by habit: talk over the agent, murmur while it speaks, pause in the middle of a sentence and go quiet. Overlapping speech is common: in the conversations analyzed by Heldner and Edlund (Journal of Phonetics, 2010), overlaps made up about 40% of all intervals between speakers. Then measure response time on every turn from the recordings.

• Talk over the greeting - state your request before the greeting ends. The agent should stop and deal with the request.
• Cut in mid-sentence - interrupt a long answer and time how long the agent's audio continues after your first word (row I-01).
• Murmur "uh-huh" - the agent should keep going (row I-02).
• Pause mid-sentence - say "I'd like to book for" and wait before you name the day. The agent should not answer the half sentence.
• Stay silent - say nothing after a question. The agent should ask again, then end the call as designed if the silence continues (row I-03).

If your framework exposes its turn-taking settings, write them next to these cases as the expected behavior. LiveKit Agents, for example, documents a default interruption min_duration of 0.5 seconds ("Minimum speech duration to register as an interruption") and a default endpointing min_delay of 0.5 seconds ("Minimum time after detected silence before the turn closes"). With the first of those defaults, an interruption registers only after half a second of speech, so set a stop-time threshold longer than the setting, or measure it from the moment the setting has passed.

Whether the agent stops, what happens to the interrupting words and the endpointing trade-off are covered in barge-in testing for voice agents.

Exit criterion - each interruption and silence case has a written expected behavior and a result, and you have a P50 and a P95 response time with their clock points.

Step 6: Test Keypad Input, Transfers and Voicemail

Test telephony from outside the agent: press the keys, answer the destination phone, let a transfer ring out, play a voicemail greeting and hang up mid-action. For each case, compare what the caller hears, what the far end hears and what the call record says.

Keypad input. ITU-T Recommendation Q.23 (1988, still in force; the wording is in the PDF on that page) defines a key press as a signal "composed of two frequencies emitted simultaneously when a button is pressed", known as dual-tone multi-frequency (DTMF) signaling. On an internet call the same press can travel as tones inside the audio or as a separate telephone-event packet (RFC 4733, 2006), so find out which form your provider delivers and test that one.

• Press a valid key while a prompt is still playing. The prompt should stop, and the call should go where that key leads.
• Enter a long number once slowly and once fast, then let the agent read it back (row T-01).
• If a menu sits in front of the agent, reach the agent by key and by speech, then press an invalid key and let one prompt time out. Menu coverage is the subject of IVR testing.

Transfers. RFC 5589 (2009), the IETF's best current practice for call transfer, names three roles: the Transferor, the Transferee and the Transfer Target. In a voice agent test, read those as the agent, the caller and the person or queue receiving the call. The RFC does not use the words warm or cold: in this guide a cold transfer is its "blind transfer", and a warm transfer is its attended transfer, in which the Transferor first "establishes a call with the Transfer Target to alert them to the impending transfer".

RFC 5589 notes that "a successful REFER transaction does not terminate the session between the Transferor and the Transferee", and RFC 3515 (2003) has the receiving side return "a 202 Accepted response" and report the outcome afterwards through NOTIFY messages. On a hosted platform the equivalent is a transfer tool that returned success: it shows that the request was accepted, and only the destination phone tells you that the call arrived.

• Trigger each hand-off condition in the contract without asking for a person (row H-01).
• Answer the destination phone yourself and note what you are told about the caller (row T-02).
• Leave the destination to ring out, then repeat with it busy (row T-03).
• Hang up as the caller right after confirming a booking, then call back (row T-05). Dropped calls and the transfer window are covered in voice agent interruption testing.

Voicemail and slow answers. An agent that places calls has to decide whether a person or a machine picked up. Twilio's Answering Machine Detection documentation says "it's possible that AMD will not always return the right answer" and names both errors: "False Machine (detected machine, actually human)" and "False Human (detected human, actually machine)". Test both with a phone you control:

• Have the agent call a phone you control and let it ring out to voicemail, once with a very short recorded greeting and once with a long one (row T-04).
• Answer a test phone late, say "hello", pause and say "hello" again. The agent should speak to you, and should not leave a voicemail-style message or hang up.

The first seconds of a call the agent places, and whether it reached the right person, are covered in inbound vs outbound phone agent testing.

Exit criterion - for every telephony case you have what the caller heard, what the far end heard and what the call record says, and the three agree with the contract.

Step 7: Verify Every Action in the Calendar, CRM or Database

Verify each action where it lands. "You're booked for Wednesday at 10:30" is the agent's claim, and the calendar entry is the result. τ-Voice scores its tasks the same way: success is "deterministically evaluated by comparing the end state of the environment (e.g., database records) against a gold standard".

• When you can see tool calls - assert that the right tool was called once, with the right date, time and patient ID, and that it returned success. Check the record as well, because a tool can be called and still fail to write.
• When you cannot see them - query the system of record before and after the call. The difference should be exactly one new or changed entry with the scripted values, and nothing else. Test accounts keep that difference readable.
• Retries - say yes twice, or slow the booking tool down so the agent tries again. The calendar must hold one entry.
• Tool errors - take the slot from a second session after the agent offers it, or point the test agent's calendar tool at a stub that returns an error (row S-01). The calendar must hold no partial entry.
• Out-of-scope questions - ask for something the agent has no source for, and check every statement in the transcript against the knowledge source (row O-01).

Exit criterion - every test case that changes something has a before-and-after check in the system it changes, and a retry or a failed tool leaves one entry or none.

Step 8: Score Each Test Call and Set the Pass Mark

Score every call on outcome, captured details, hard rules, response time and interruption handling, and pass it only when all of them pass. The suite's pass mark has two parts: the thresholds from your contract, and a baseline you record on your own agent.

• Rule checks first - the end state, the tool that was called, the digits captured, the read-back and the response times can all be checked without a model.
• A model as judge for the rest - tone, clarity and staying in scope need judgment. Bavaresco et al. tested 11 LLMs on 20 datasets with human annotations and concluded that "LLMs should be carefully validated against human judgments before being used as evaluators" (arXiv 2406.18403, accepted to ACL 2025). Label a sample of your own calls by hand and compare the judge's verdicts with yours before you rely on it.
• Repeat runs - the same scenario can pass on one run and fail on the next. A study of coding agents collected 60,000 trajectories and found that single-run pass rates "vary by 2.2 to 6.0 percentage points depending on which run is selected" (arXiv 2602.07150). No source gives a universal repeat count, so run each scenario several times and report a pass rate.
• Report the spread - Evan Miller's paper on evaluation statistics suggests "reporting the standard error of the mean alongside (beneath) the mean when reporting eval scores" (arXiv 2411.00640). Do not act on a difference between two builds that is smaller than the run-to-run spread.

The guide to LLM-as-a-judge covers building a judge, and pass@k vs pass^k covers computing a rate from repeated runs.

SAMPLE DATA: 8 invented calls to a fictional appointment-booking voice agent, not real measurements.

call     condition                  outcome            slots  slowest  barge-in  rules     result
call-01  quiet line                 booked ok          2/2    1020 ms  none      ok        PASS
call-02  Scottish accent            booked ok          2/2    1130 ms  none      ok        PASS
call-03  street noise               booked ok          1/2    1260 ms  none      ok        FAIL: slots (time)
call-04  caller interrupts          booked ok          2/2    1090 ms  1240 ms   ok        FAIL: barge-in
call-05  caller presses keys        booked ok          4/4    2140 ms  none      ok        FAIL: slowest turn
call-06  caller changes their mind  rescheduled ok     2/2    1210 ms  none      1 broken  FAIL: rules
call-07  asks for a human           transferred ok     1/1    960 ms   310 ms    ok        PASS
call-08  long silence               refused ok         2/2    1340 ms  none      ok        PASS

Suite: 8 calls, 38 agent turns
  task success     7 of 8 calls (87.5%), target at least 90%
  slot accuracy    lowest call 50%, target 100% on every call
  latency          p50 940 ms, p95 1880 ms, target p95 at most 1500 ms
  barge-in stop    worst 1240 ms, target at most 500 ms
  rule violations  1, target 0

GATE: FAIL
  - task success 87.5% is below 90%
  - call-03 captured time 16:50, expected 16:15
  - p95 latency 1880 ms is over 1500 ms
  - call-04 kept talking 1240 ms after the caller interrupted, limit 500 ms
  - call-06 confirmed the booking without reading the date and time back

// Percentile by the nearest-rank method: sort the values and take the one at position
// ceil(p / 100 * n), counting from 1. No interpolation, so the answer is always a latency
// that really happened. With fewer than 20 values the 95th percentile is the largest one.
function percentile(values, p) {
  const sorted = [...values].sort((a, b) => a - b);
  return sorted[Math.ceil((p / 100) * sorted.length) - 1];
}

Run the suite several times on the build you plan to launch, or on the release that is live today if the agent already takes calls. The average of each number is your baseline, and the range across those runs is the run-to-run spread.

A hard rule fails a scenario if it breaks on any run, and every other check is reported as a pass rate over the runs. From then on, the gate has two parts: the contract thresholds, which every release has to meet, and the baseline, which no change may fall behind by more than that spread.

Exit criterion - the suite returns one PASS or FAIL with its reasons, from thresholds written down before the run.

Step 9: Automate the Test Cases With Simulated Callers

A simulated caller is a program, usually a language model given a goal and a persona, that holds the conversation in place of a person. How the simulated caller reaches the agent decides what a run can catch:

• In text - the caller types. This catches logic, tool use and broken rules, runs fast enough for every commit and misses everything about hearing and timing.
• In audio, against the endpoint - the caller speaks over a web or WebSocket connection. This adds speech recognition, speech output, interruptions and response time, and misses the phone network.
• Over a phone call - the caller dials the number. This adds carrier audio, the keypad, transfers and voicemail, and is the slowest of the three.

The voice platforms ship their own test tools:

• LiveKit Agents - unit tests, in which "you run your agent turn by turn and assert on its responses", and Agent Simulations (in beta) with "an LLM-driven simulated user that follows a scenario from start to finish", in a text mode and an audio mode.
• Pipecat - Pipecat Evals, in a text mode and an audio mode: "You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent".
• Vapi - Evals, which check the agent's next response "using exact matching, a pattern, or an AI judge", and Simulations, in which an AI tester "acts as the caller and adapts during a complete conversation".
• Retell AI - an LLM Playground, Simulation testing, Web call testing and Phone call testing, the last to "validate telephony: carrier audio, DTMF, and transfers".
• ElevenLabs - simulation tests, next reply tests and tool call tests. Check whether they run on audio before you rely on them for Step 4 or Step 5.

Each of these tests agents built on its own platform. When your agents run on more than one stack, or you want a result that does not depend on the platform's own view of the call, test from outside by dialing the number.

Before you file a failed run as an agent bug, read its transcript and listen to its recording. A simulated caller can wander off its script, mishear the agent or hang up early, so sort each failure into agent, simulator or environment, and fix the scenario when the simulator was at fault.

One no-code session is written up in how to test a voice agent without code, and the products are compared in AI voice agent testing tools.

Exit criterion - every row of the table that a simulated caller can perform runs without a person, and every failed run has been sorted into agent, simulator or environment.

Step 10: Rerun the Test Suite on Every Prompt, Model and Telephony Change

Rerun the suite whenever something that shapes the agent's behavior changes, compare the result with the baseline from Step 8 and block the change when the gate fails. These changes should trigger a run:

• A prompt edit, however small.
• A new model or model version, including the speech recognizer and the voice.
• A tool added, removed or given a new schema.
• A knowledge source updated.
• A turn-taking or telephony setting changed, such as endpointing, numbers, trunks or transfer targets.
• A provider update you did not choose, which is why the suite also runs on a schedule.

The loop is the same for each of them:

• Run the text suite on every commit.
• Run the audio and phone suites before each release and on a schedule.
• Compare every scenario with its baseline result.
• Fail the change when a hard rule breaks, a contract threshold is missed or a number falls behind its baseline by more than the run-to-run spread.
• Accept new baseline numbers only as a deliberate decision.

Every failed production call becomes a test case. Write down the caller, the condition and what they did, write the pass criterion, and confirm that the case fails on the build that produced the call before you fix anything. The guide to AI voice agent regression testing covers what regresses in a voice agent and which metrics to track.

Exit criterion - no prompt, model, tool or telephony change reaches production without a suite run compared with the baseline.

Step 11: Load Test the Voice Agent Before Launch

Load is a separate test with its own pass rule: the question is how many calls at once the agent, the phone trunk and the model provider can carry before calls fail or slow down.

• Write down the concurrent-call limit of your voice platform account, the channel limit of your phone trunk and the rate limits of your model and speech providers, and test an environment that has the same limits as production.
• Choose the concurrency levels to test, with your expected peak among them.
• Make every concurrent call a simulated caller that holds a whole conversation (Step 9), because a call that connects and stays silent puts no load on the recognizer, the model or the voice.
• Ramp up to each level, hold it for a set period and measure only during the hold.
• At each level, record how many calls connected, how many ended normally, what a caller over the limit heard (a queue message, a busy tone or silence) and the P50 and P95 response time.
• Decide before the run what counts as breaking.

TestMu AI's phone agent documentation gives one written example of such a rule: the breaking point is the lowest concurrency group where the success rate falls below 80%, or where P95 response latency exceeds twice the P95 of the lowest group.

A load test says nothing about whether the conversations were right, so keep running the scored suite beside it.

Exit criterion - you know the concurrency at which reliability or response time breaks, and it is above your expected peak.

Step 12: Monitor Production Calls After Launch

Launch in stages, score real calls with the checks you used before launch, and alert on change from the baseline.

• Staged launch - start with a share of calls, one task or limited hours, with the hand-off to people staffed, and widen when the numbers hold.
• Same checks on real calls - sample production recordings and score outcome, captured details, hard rules and response time exactly as in Step 8, and count the tool calls that failed.
• Business numbers beside test numbers - track the containment rate (calls resolved without a person) and the hand-off rate next to the pass rate. No neutral source for a good containment rate was found, so read it against your own baseline. A rise in hand-offs or repeat calls can show a failure the suite does not cover.
• Alerts on change - alert when the pass rate, a response-time percentile, the failed tool calls or a business number moves away from its own baseline.

For products that do this, see the comparison of voice agent monitoring tools.

Exit criterion - production calls are scored on a schedule you set, and every failure found there has become a row in the test case table.

Voice Agent Tests You Can Run on TestMu AI

With TestMu AI's Agent Testing you can test chat, voice, phone (inbound and outbound), video and image agents. For a voice agent you pick the Voice type or the Phone Caller type, which correspond to the last two modes in Step 9:

• Voice - you connect the agent's endpoint over REST or WebSocket. The platform "holds a spoken conversation with your agent, transcribes the responses, and scores the interaction" on 9 quality metrics, among them hallucination detection, completeness, context awareness and conversation flow. The evaluation runs on the transcript of the audio conversation, so the response-time and interruption figures in the list below come from the Phone Caller type.
• Phone Caller - the platform tests "by placing real telephone calls, not simulations". For inbound it calls your agent's number, for outbound your agent calls a number the platform answers, and in both a simulated caller follows a scenario and the call is scored on 30+ call quality metrics. It works with "any agent that answers a phone number, on any platform, with no code changes, SDK, or test build".

On a phone test:

• Test cases (Step 3) - you generate up to 20 inbound scenarios or up to 7 outbound ones, or create them manually, and the agent prompt is the evaluation baseline.
• Voices and noise (Step 4) - each scenario takes a voice from a library of 200+ voices "with accents and speech speeds" and one of 15 background-noise presets.
• Timing (Step 5) - the call metrics include Average Latency, AI Interrupting User and User Interrupting AI. The documentation describes these as measurements of a call and describes no way to place an interruption at an exact moment, so plan that as a separate case.
• Keypad (Step 6) - keypad inputs (0 to 9, star, pound) are captured in the transcript. The documentation describes no dedicated voicemail check, hold check or transfer procedure. Escalations are counted in the AI to Human Handoff Rate metric, so keep the destination-phone checks from Step 6 and write the voicemail and slow-answer cases as scenarios of their own.
• Tool calls (Step 7) - with the ElevenLabs, Retell or Vapi integration, each tool a scenario expects is marked Called or Not Called after every test call. The mark does not show that the tool succeeded, so the check in the system of record stays with you.
• Scoring (Step 8) - each result includes "the transcript, the call recording, pass or fail grading against your criteria, and call and audio quality metrics", and a run gets a Green (80 or more), Yellow (65 to 79) or Red (below 65) verdict. Custom metrics grade calls "on rules you write in plain language", from the transcript only.
• Load (Step 11) - performance testing, for agents that answer calls, ramps up at 5 calls per second, scores only the hold phase and reports P50 and P95 response latency. It produces no conversation quality scores.
• Monitoring (Step 12) - you can upload recorded production calls as MP3 or WAV files and have them scored with the same metrics.

For Step 10, phone suites run from the dashboard, on a schedule with preset frequencies or custom cron expressions, or from the Agent Testing CLI:

pip install agent-testing-cli
agent-testing-cli login

agent-testing-cli --project PROJECT_ID run \
  --suite SUITE_ID \
  --yes \
  --wait \
  --poll 5 \
  --timeout 1800

The --yes flag skips the confirmation the CLI asks for "because a Phone Caller suite can create real calls", and --poll and --timeout are in seconds. Exit code 1 means a completed test failed, which is what lets a pipeline job fail on it. In a pipeline, leave out the login command and set the LT_USERNAME and LT_ACCESS_KEY environment variables instead, and keep their values in the secret store of your CI/CD platform.

The Agent Testing CLI documentation covers this run command for Phone Caller suites and a separate one for Chat evaluations, and none for the Voice type. Set-up for inbound and outbound tests, custom metrics and performance testing is in the Phone Agent Testing documentation.

Note: TestMu AI Agent Testing places real calls to your voice agent with a simulated caller and returns the recording, the transcript and call quality metrics for each one. Get started on TestMu AI

Voice Agent Go-Live Checklist

Take this list to the release decision. Each line names the step that produces its evidence, and a line with no evidence behind it is your next piece of work.

• The test contract is written, and every line names what proves it (Step 1).
• Every task in the contract has completed on a real phone call, and the call log is saved (Step 2).
• Every contract line has at least one test case with a pass criterion you can observe (Step 3).
• The share of calls with every detail captured is known for each accent, noise level and line condition your callers have (Step 4).
• P50 and P95 response time are recorded with their clock points, and the interruption and silence cases pass (Step 5).
• The keypad, hand-off trigger, warm transfer, unanswered transfer, voicemail and dropped-call cases pass, with a tester on the destination phone (Step 6).
• Every action is verified in the system of record, and retries and tool errors leave one entry or none (Step 7).
• The suite meets your written thresholds over repeated runs with zero broken hard rules, the baseline and the run-to-run spread are recorded, and any model used as a judge agrees with your hand labels on a sample of your calls (Step 8).
• The automated suite runs on every prompt, model, tool and telephony change, and on a schedule (Steps 9 and 10).
• The load test's breaking point is above your expected peak (Step 11).
• Production calls will be scored with the same checks, and the hand-off to a person is staffed for launch (Step 12).
• In the United States, for calls your business places, prior express consent is on file for every contact dialed, since the FCC's ruling FCC 24-17 treats AI-generated voices as artificial under the TCPA (row R-01), and your own policy on disclosing an AI-generated voice is written and tested (row R-03).
• In the European Union, the caller is told at the first interaction that they are speaking with an AI system, as Article 50 of the AI Act asks of providers (row R-02).
• For card payments, the security code is absent from stored recordings and transcripts, following the PCI Security Standards Council's telephone guidance (row R-04).

If you are working out how to test a voice agent for the first time, start with the Step 1 contract for the one task your callers need most, then place the first manual call against it.