Next-Gen App & Browser Testing Cloud

Trusted by 2 Mn+ QAs & Devs to accelerate their release cycles

How Do We Test an IVR-to-Voice-AI Migration?

By Anupam Pal Singh. Last reviewed November 14, 2026.

Test an IVR-to-voice-AI migration by running the same scenarios, with the same caller goals and pass criteria, against the old IVR number and the new voice AI number, then comparing the results. Cut over when the voice AI line completes at least as many tasks, transfers callers correctly, meets your latency targets and invents no details.

The phone agent testing docs describe the pieces this needs: real telephone calls to inbound lines and IVR flows, a simulated caller that follows each scenario, and the same 42 call quality metrics on every call.

For the old menu alone, see our IVR testing guide; for a full plan for the new agent, see how to test a voice agent.

What Must Stay the Same Between the IVR Run and the Voice AI Run?

The comparison holds only if the dialed number, plus the keypad setting the IVR needs, is all that changes:

Caller goals: the old menu's paths and the top reasons people call. For a Twilio IVR, a separate IVR Agent maps its menus, branches and dead-ends in an IVR System Map (see the Twilio IVR guide); each path that ends in self-service or a transfer is a goal the voice AI line must also complete.
Callers and test data: the same personas, voices, background noise, region, test profiles and iterations.
Pass criteria: the same success criteria in each scenario, with both suites in one Phone Caller agent and its metric settings left unchanged between the runs.
Keypad input: if the old line expects DTMF tones, turn on DTMF / IVR Navigation with menu instructions on the IVR copy of each scenario so the simulated caller works through keypad menus; keypad inputs are captured in the transcript. Leave it off on the voice AI copy unless the new line keeps a keypad menu.

How Do You Run the Same Scenarios Against the IVR and the Voice AI Line?

Follow the documented inbound workflow, with one agent and two suites:

Create one agent: choose Phone Caller Agent as the agent type and Inbound (Bot receives calls) as the call type.
Add context: on the Prompt tab, describe the facts, policies and rules both lines must follow. Leave out the old IVR's keypad script, because Behavior adherence and Faithfulness grade the agent against its instructions and script; write the caller goals from the menu map into the scenarios instead.
Build the scenarios: generate up to 20 or write them, then add the IVR copies with DTMF / IVR Navigation on.
Register both numbers: add the IVR's number and the voice AI line's number on the Phone Numbers tab.
Create two suites: one runs the IVR copies against the IVR number, the other runs the voice AI copies against the voice AI number, with every other setting identical.
Run both: the Confirm Run dialog lists every real call before anything is dialed.
Compare the roll-ups: each check shows its pass rate across the suite, leaving out calls where it did not apply, and scored checks add the average, median and 90th percentile.

Which Results Should You Compare Before Moving Callers to Voice AI?

Compare the checks that mean the same thing on a menu and a conversation. Targets are the documented defaults, editable per project.

Check	What it measures	What to compare
Task completed	Whether the agent completed what the caller asked for	Pass rate, and which caller goals each line failed
Expected result reached	Whether the call met the scenario's success criteria	Criteria each line missed, on the Validation Results tab
Containment rate, AI to human handoff	Whether the agent kept the call or passed it to a person	Handoffs that match each scenario
First call resolution	Whether the issue was solved on this call	Pass rate where no transfer is expected
Not early termination	Whether the agent ended the call too soon	Early endings the IVR did not have
Time to First Word, Average Latency, Max latency, Dead air	Waits for the first words, for each reply and in silence	Median and 90th percentile against 1.5 s, 2.5 s, 3 s and 5 s
Mean opinion score	A predicted 1 to 5 rating of call audio clarity	Both lines at or above 3.5

Which Checks Apply to the Voice AI Line but Not the Scripted IVR?

A menu IVR plays fixed prompts, while a voice AI agent generates each reply (see IVR vs IVA). The Twilio IVR guide says hallucination, grounding and safety metrics do not apply to a scripted flow, so read these checks on the voice AI suite only and compare the two lines check by check, not on call verdicts:

No invented details: fails when the agent states a name, number, date, status, booking or policy that nothing on the call or in its instructions supports.
Response consistency: catches the agent contradicting itself, such as quoting a balance of $120 and later $210.
PII protection: fails when the agent reads out a secret in full, such as a card number or a PIN.

The built-in PII protection check is one of the safety metrics the guide leaves off for a scripted flow, yet a scripted IVR can still read a full card number aloud. Write that rule, and rules the menu enforced by design, such as asking for a PIN before changing an address, as custom metrics attached to both copies of the scenario; a new custom metric is report-only until you turn it on in Metric Config.

Where Does TestMu AI Fit in an IVR-to-Voice-AI Migration?

With Agent Testing you can test chat, voice, phone (inbound, outbound and IVR), video and image agents. Before the new agent has a phone number, the Voice Agent type can test it over its voice endpoint through an endpoint profile (see the voice agent docs). It scores 9 quality metrics rather than the 42 phone metrics, so use it as an early check, then compare both lines with a Phone Caller agent once the new number is live.

Frequently Asked Questions

Can production call recordings from the old IVR be a test baseline?

Yes. Recording analysis scores batches of recorded production calls (MP3 or WAV) with the same metrics as live test calls, without placing new calls. Score a batch from the IVR before cutover and one from the voice AI line after it, so the comparison covers real callers, not only your scenarios.

How do you test calls the new voice AI agent should transfer to a person?

For each scenario where a person should take the call, turn on Transfer to human agent in the scenario editor and write the transfer into its success criteria. On those scenarios, the docs say AI to human handoff should read Yes, Containment rate is not counted, First call resolution shows Handoff scenario, and Accuracy rates how cleanly the agent handed the caller over. Warn any staffed queue first: these are real calls.

How do you load test the new voice AI line before cutover?

Run a performance test on the voice AI suite. It places real calls from the suite's scenarios, holds a target number of concurrent calls for 1 to 120 minutes (5 by default) and reports load delivery, reliability and P50 and P95 response latency. Set the target near the IVR's peak concurrent calls, up to your available parallels. It does not score conversation content.

What if the new voice agent runs on ElevenLabs, Retell, Vapi or Agentforce?

Connect it in the agent's Integrations tab before you fill in the Prompt tab: syncing imports the voice agent's prompt into the Prompt tab both suites share, along with its tool catalog. Each test call then shows which expected tools the agent called; read those results on the voice AI suite only, since the integration is connected to the voice AI agent, not the IVR.

How do you keep testing the voice AI line after cutover?

Put the voice AI suite on cron-based scheduled runs, which support IANA time zones, pause and resume, and a run history, so the same scenarios keep dialing the live number. Keep scoring batches of production recordings too, because real callers raise requests that a fixed scenario set may not cover.