How to Test a Retell AI Voice Agent: Step-by-Step Guide

How Retell AI Agents Are Built

Retell's introduction docs present one platform for voice and chat agents that bundles telephony, prompts, tools and call analytics. Its agent type guide lists two current types: a single prompt agent, where one prompt steers the whole call, and a Conversation Flow agent, which splits the call into nodes.

The older multi-prompt type is being retired. Retell's multi-prompt deprecation notice says the API stops creating multi-prompt agents on December 15, 2026 and stops edits to existing ones on February 1, 2027. Any multi-prompt agents left after that are converted to Conversation Flow agents, and Retell warns that a converted agent may not handle the same call the way the original did.

Retell's node reference describes a Conversation Flow as nodes joined by edges. Conversation nodes hold dialogue without tool calls, Subagent nodes hold dialogue and can call tools, and Function nodes run one tool as soon as the call enters them. Retell's global node docs add that a global node can be reached from anywhere in the flow.

Teams that run their own model connect it as a custom LLM, and Retell's custom LLM overview says Retell then opens one WebSocket to your server per call while keeping telephony, transcription, turn-taking and speech synthesis. Retell's LLM API reference also accepts a speech-to-speech model in place of a text model.

Retell's web call and telephony docs describe two ways a call reaches a voice agent: a web call in the browser through Retell's Web SDK, with no phone number, or a phone call on a number bought from Retell or on your own carrier's number over SIP trunking, with guides for Twilio, Telnyx and Vonage among others. Each number on the Phone Numbers page gets an inbound agent and an outbound agent, each set to the version you choose.

The build choice decides which tests are possible at all. The table pulls the testing consequences of each choice from Retell's agent type guide, Conversation Flow docs, testing overview, custom LLM overview and chat agent docs.

To see how Retell's builder and telephony options compare with two other voice platforms, read the Vapi vs Retell vs Bland AI comparison.

How Retell AI Agents Fail in Production

Each failure below traces back to a documented Retell setting or limit. Most surface only with real callers, because accents, background noise, interruptions and real call volume rarely reach a text test or a quiet demo.

• Latency spikes break turn-taking - Retell's Get Call reference reports each call's end-to-end latency, from when the caller stops talking to when the agent starts, plus separate LLM, transcription, text-to-speech and knowledge base figures at P50 to P99. A slow tool endpoint or knowledge base lookup mid-turn stretches that gap, so read the figures per call rather than trusting a demo that felt fast.
• Tool calls fail mid-call - When the booking or CRM API behind a function times out or returns an unexpected shape, the agent can stall, answer with stale data or confirm an action that never ran. TestMu AI's Retell testing docs describe the last case: a transcript in which the agent confirms a cancellation while the cancel tool shows Not Called. A confirmation like that is a hallucinated action, one of the forms AI hallucinations can take.
• Interruption handling misfires on real audio - In Retell's background noise guide, interruption sensitivity runs from 0 to 1, and the higher the value, the more easily the agent is interrupted; the API defaults to 1, and new blank voice agents in the dashboard start at 0.9. Set it too high and a cough or a background voice cuts the agent off; set it too low and a caller who means to interrupt gets talked over, so re-test after every change to it or to the denoising mode.
• Knowledge base and prompt drift after updates - Retell's Node KB changelog of September 22, 2025 notes that knowledge bases "could only be set globally" before node-level assignment, which narrows retrieval for "more accurate answers" and "lower latency". Every content update to that knowledge base can still change what the agent says on its node with nobody touching the prompt.
• Transfers fail silently - Retell's transfer call docs describe a cold transfer, where the agent drops off as soon as the call goes to the destination, and a warm transfer, where it stays on the line and, when the destination queues calls, waits up to a Wait Time for Agent Answer of 30 seconds by default for a real person. Showing the caller's own number needs caller ID override from your telephony provider or the transfer fails, and an agentic warm transfer that times out is either cancelled as failed or bridged anyway, depending on its Action on Timeout.
• Keypad input breaks on timing - Caller digit entry ends at a digit limit, a termination key or a timeout of 1,000 to 15,000 ms after the last digit, per Retell's DTMF input settings, so a timeout set too short cuts a slow caller off mid-number.
• Phone menu navigation breaks on timing - When the agent dials into another company's phone menu, Retell's Press Digit docs set a Pause Detection Delay, the time the agent waits after the menu goes quiet before it presses, at 1,000 ms by default within a 0 to 5,000 ms range.
• Concurrency limits throttle call volume - Per Retell's concurrency docs, the 20-call default applies per workspace, and inbound, outbound and web calls all draw from that one pool, test calls included. At the limit an outbound call is rejected unless concurrency burst is enabled, and an inbound call on a number with no fallback number configured ends with the disconnection reason concurrency_limit_reached.

Retell's Built-In Testing Tools and Where Each One Stops

Retell's testing overview lays out a workflow of four steps: build and debug, lock in a regression suite, validate the voice, then confirm telephony. Each step uses a different tool, and only simulation testing grades a run.

• LLM Playground - The Test LLM tab of the agent's test panel, where you type to the agent yourself or let an AI-simulated user talk to it, then check each turn's tool calls and node transitions. You can edit a user turn, rerun an agent response or replay the chat, but the Playground does not score the run.
• Simulation testing - A saved test case holds a prompt describing the simulated caller, its success criteria, preset dynamic variables, custom function mocks and the model that plays the caller. Retell's simulation testing docs say all criteria are judged together against the transcript, so one unmet criterion fails the run, and a run times out at 10 minutes or is cut off past 400 utterances.
• Batch testing - Selected cases run together, each exactly once, and land in Batch Testing History, which lists batches for one agent version at a time. Over the API you start a batch, poll until its status is complete and read pass_count, fail_count, error_count and total_count, which is enough to gate a deploy in CI.
• Conductor - Retell's AI copilot in the dashboard drafts test cases from the transcripts and tool history of calls you point it at, or writes a starter suite for an agent with no call history, then runs the batch and posts the pass rate and the details of every failed case in the panel. A tool call that matches none of its mocks falls through to the real tool, so review the mocks before the run.
• Web call testing - A browser call with real audio that exercises latency, turn-taking and interruptions, needs no phone number and can test a draft without publishing it. Transfers to a phone number and SMS fail on a web call, and Retell counts the test as a real call for billing and concurrency.
• Phone call testing - A real call on one of your numbers checks keypad input and Press Digit, warm and cold transfers, voicemail and IVR detection, SMS delivery, and audio quality and latency. Newer workspaces need identity verification before any outbound call, test calls included, and any draft version can be attached to a number without publishing it.
• Post-call analysis and AI QA - Post-call analysis fills call_summary, user_sentiment, call_successful and in_voicemail plus custom Boolean, Text, Number or Selector fields, and user_sentiment comes from the transcript, not from tone or pitch. AI QA samples real calls, scores them against rules and metrics you configure and points to per-call problems such as hallucinations, overlapping speech, knowledge base accuracy, sentiment and tool usage, with transcript evidence for each.

Because simulation is a text exchange, it never shows how the agent hears, sounds or keeps pace on a call. Inside a simulation, transfers and SMS are always faked, while any custom function, code, calendar or MCP tool without a mock reaches its live production endpoint, and MCP tools skip mocks entirely. Since one LLM plays the caller and another grades the run, Retell's simulation docs advise rerunning a batch before you act on one failure.

Retell's tools are enough early in a build. Text simulation is where Retell's pricing docs send most checks, and with function mocks it is the most direct way to force a tool result: Retell's Conductor docs show a case in which check_availability comes back with no open slots and the agent must offer a callback rather than hang up. Neither the phone agent docs nor the Retell integration docs at TestMu AI describe mocking a function result, so keep mocked edge cases in Retell's suite.

Add an external real-call layer when you need to prove the voice and phone path under realistic callers, test a custom LLM or speech-to-speech agent beyond manual calls, load-test an inbound line before launch, or score calls on one set of metrics across Retell and another platform.

Four Dimensions to Test Before a Retell Agent Goes Live

Each failure mode above maps to one of these dimensions, and all four belong in the test run for every build.

• Task success - Was the caller's goal met by the end of the call? An appointment agent that ends the call politely without booking anything is a failure the call log will not show you unless you check the outcome, not just the transcript's tone.
• Conversation quality - Did the agent stay on topic, retain context from earlier turns, and respond completely? A caller who repeats their account number because the agent forgot it three turns later will not call back.
• Safety - Does the agent avoid inventing policies or pricing it was never given, and does it hold up when a caller tries to extract information it should not share or push it off its script? A hallucination rate counts invented answers like these, and the guide to AI hallucination rate compares how public benchmarks count them.
• Resilience - Does the agent survive the conditions a real phone network throws at it: background noise, a caller who interrupts, a dropped function call, a transfer that has to retry?

This same four-part split applies to any conversational AI system, and the deeper mechanics of scoring conversation quality specifically are covered in conversational AI testing. For a phone agent, resilience also covers the telephony layer (latency, interruption timing and call handoffs), and how to test AI calling agents works through that layer for phone agents on any stack.

Retell Agent Test Checklist: Flows, Tools, Keypad and Transfers

Generic voice agent cases such as greetings, accents, background noise and handoffs are covered with pass criteria in how to test a voice agent. Add these cases, which come from how Retell builds and connects agents.

• Every node in the flow - Map each scenario to the nodes it should pass through and the End node it should finish on, until the suite reaches every node; TestMu AI's Retell testing docs list a call that ends on the wrong node as a common trip-up. Retell evaluates global nodes only after a user turn, so script the caller line that should trigger each one.
• Tool results beyond success - For each function, cover a success, an empty result, an error and a timeout, and check that the agent never confirms an action whose tool did not run. In Retell simulation, give every mock a non-empty result, because a blank mock is skipped and the real function runs.
• Dynamic variables - Check that names, account numbers and order IDs passed as dynamic variables reach the prompt and the tool arguments. Retell injects a tag's dynamic variables into every live call on that tag, so keep test-only values off the production tag.
• Caller keypad entry - Enter digits slowly, quickly and with the termination key, and check how the agent responds when entry hits the digit limit or the timeout.
• Press Digit through phone menus - For an agent that dials into another company's phone menu, test menus that pause at different points against the Pause Detection Delay, because "DTMF fires early" is on the same list of common trip-ups.
• Transfers on a phone call - Run every warm and cold transfer on a real call, since a web call cannot reach a phone number. Check the destination, what the person who answers hears, the wait when the destination queues, and caller ID override with your carrier.
• Voicemail and silence - On outbound calls, check that the agent detects voicemail and takes the action you configured, and that post-call analysis sets in_voicemail. On every call type, script a caller who goes quiet and check that the agent prompts again and then ends the call politely.
• Calls near the concurrency quota - On a workspace that carries no live traffic, push a burst of calls close to the quota and confirm that the fallback number answers once inbound slots run out, since every call in the workspace shares the same pool.
• Chat and voice parity - If the chat agent was converted from the voice agent, run the shared scenarios on both, since Retell's chat agent docs describe the result as a new agent and the voice agent as unchanged.
• Personas beyond the happy path - Retell's simulation docs ask for edge-case callers, for example one who loses their temper, one who will not confirm who they are and one who switches requests halfway through. AI agent simulation covers how to build and size a persona set.
• Personal data in recordings - If PII scrubbing is enabled, play the scrubbed recording from the Get Call API and confirm that the categories you chose to scrub, such as card numbers, dates of birth or addresses, are gone from it.

scenario: Booking request when no slot is free
agent: Conversation Flow, inbound, version V3
caller: busy parent, changes the day once mid-call
line: call center noise, response timing 1.5 s
caller_data: {caller_name}, {preferred_day}
expected_tools:
  - check_availability   # must show Called
agent_must:
  - read the requested day back before checking
  - say that no slot is free on that day
  - offer another day or a callback
agent_must_not:
  - confirm any booking
  - call book_appointment
pass_if:
  - every must and must-not rule holds
  - book_appointment absent from AUT Tool Calls
  - Average Latency at most 1000 ms

Which Metrics Should You Track for a Retell Agent?

Retell's call fields and AI QA metrics describe what happened on calls that ran on Retell, and TestMu AI's call metrics add published thresholds to grade test calls against. Field names come from Retell's Get Call reference, post-call analysis docs and AI QA terminology; thresholds come from TestMu AI's inbound phone agent testing docs.

How to Test a Retell AI Agent With Retell's API

A homegrown harness places a call, follows its webhooks, pulls the finished call record and scores it, and Retell's API covers the first three steps.

• Place the call - Send a POST to Retell's create-phone-call endpoint (client.call.create_phone_call in the Python SDK) with a from_number you bought from or imported to Retell and a to_number, both in E.164 format; a Retell-purchased from_number can only call US numbers. override_agent_id and override_agent_version swap the agent or its version for this one call without rebinding the number, and an idempotency_key lets you retry the request for up to an hour without placing a second call.
• Follow the webhooks - Retell's webhook events for a voice agent default to call_started, call_ended and call_analyzed, while transcript_updated and the transfer events (transfer_started, transfer_bridged, transfer_cancelled, transfer_ended) arrive only if you list them in the agent's webhook_events field. Retell waits 10 seconds by default (the agent's webhook_timeout_ms) for a 2xx response and retries up to 3 times, so make the consumer idempotent, and expect a call that never connected (dial_failed, dial_no_answer, dial_busy) to skip call_started but still send the other two.
• Pull the call record - call_ended carries every call field except call_analysis, and call_analyzed adds it. GET /v2/get-call/{call_id} returns the transcript, transcript_with_tool_calls, recording_url, the latency breakdown and disconnection_reason.
• Score and repeat - Grade each call against your rules, rerun the scenario across personas, voices and interruption patterns, and judge it on its pass rate across runs, the same advice Retell gives for its own simulations.

Step 1 as a Node.js request, with placeholder IDs and numbers:

// Place one test call through Retell's API
const url =
  "https://api.retellai.com/v2/create-phone-call";

const res = await fetch(url, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.RETELL_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    from_number: "+14155550100",
    to_number: "+14155550199",
    override_agent_id: "agent_booking_test",
    override_agent_version: 3,
    idempotency_key: "nightly-2026-10-17-case-07",
    metadata: { suite: "no-free-slots", run: "nightly" },
  }),
});

const call = await res.json();
// call_status starts as "registered"
console.log(call.call_id, call.call_status);

Retell retires parts of its API: its deprecation notice records that legacy list endpoints, among them the ones for batch tests and test runs, were removed on June 15, 2026. A harness that calls a removed endpoint or reads a renamed field fails in a way that looks like an agent bug, so before a nightly run, check what the sample above and a batch gate rely on against Retell's published OpenAPI file. The script needs Node.js 18 or newer and the js-yaml package (npm install js-yaml), but no Retell API key. Saved as check-retell-spec.mjs, it runs as an ES module, sets exit code 1 when anything it checks is missing, and names anything Retell has marked deprecated:

// Check the Retell API surface a test harness relies on
import { load } from "js-yaml";

const url = "https://docs.retellai.com/openapi.yaml";
const spec = load(await (await fetch(url)).text());
const { info, paths, components: { schemas } } = spec;
const ref = (s) =>
  s?.$ref ? schemas[s.$ref.split("/").pop()] : s;
const props = (s) => ref(s)?.properties ?? {};
const body = (path) => props(paths[path]?.post
  ?.requestBody?.content?.["application/json"]?.schema);
const statuses = (name) => Object.fromEntries(
  (props(schemas[name]).status?.enum ?? [])
    .map((value) => [value, {}]));
const flagged = (v) => v?.deprecated === true ||
  ["get", "post", "patch", "delete"]
    .some((m) => v?.[m]?.deprecated === true);

const checks = [
  ["endpoints", paths,
    ["/v2/create-phone-call", "/create-batch-test",
      "/get-batch-test/{test_case_batch_job_id}"]],
  ["create-phone-call", body("/v2/create-phone-call"),
    ["from_number", "to_number", "override_agent_id",
      "override_agent_version", "idempotency_key",
      "metadata"]],
  ["call response", props(schemas.V2CallBase),
    ["call_id", "call_status"]],
  ["create-batch-test", body("/create-batch-test"),
    ["test_case_definition_ids", "response_engine"]],
  ["batch job", props(schemas.TestCaseBatchJob),
    ["test_case_batch_job_id", "status", "pass_count",
      "fail_count", "error_count", "total_count"]],
  ["batch status", statuses("TestCaseBatchJob"),
    ["in_progress", "complete"]],
  ["run status", statuses("TestCaseJob"),
    ["pass", "fail", "error"]],
];

console.log(info.title, info.version,
  info["x-retell-spec-revision"] ?? "");
for (const [name, have, want] of checks) {
  const gone = want.filter((k) => !(k in have));
  const old = want.filter((k) => flagged(have[k]));
  if (gone.length) process.exitCode = 1;
  console.log(`${name}: ${want.length - gone.length}` +
    `/${want.length} found` +
    (gone.length ? `, missing ${gone.join(", ")}` : "") +
    (old.length ? `, deprecated ${old.join(", ")}` : ""));
}

Run on October 17, 2026 with Node.js 25.5.0 and js-yaml 5.5.0 against Retell's spec revision 2026-10-16-5d9ba01, it found everything it checks and nothing marked deprecated:

$ node check-retell-spec.mjs
Retell SDK 3.0.0 2026-10-16-5d9ba01
endpoints: 3/3 found
create-phone-call: 6/6 found
call response: 2/2 found
create-batch-test: 2/2 found
batch job: 6/6 found
batch status: 2/2 found
run status: 3/3 found

A batch is done when its status reads complete, and a run that ends in error has no verdict at all, so a gate that checks only fail_count would let it through. Gate on pass_count equal to total_count instead.

Steps 1 to 3 are plain HTTP calls, and step 4 at volume is where homegrown harnesses stall, because someone has to supply realistic callers with varied voices and line noise, place the calls in parallel and grade intent and policy rather than keywords in the transcript.

How to Test a Retell Agent With TestMu AI, Step by Step

TestMu AI's Agent Testing places real phone calls to a Retell agent with simulated callers, or reserves a number for an outbound agent to call, and scores each conversation with no SDK, code change or test build on the agent's side. The product overview is on the Retell AI testing page, and the full setup is in the Retell agent testing docs.

Before you start, have a published agent in your Retell account, a Retell number or one connected over SIP trunking for phone tests, the prompt, PRD or knowledge base that scenarios will be generated from, and agent-testing permissions in your TestMu AI workspace.

• Connect the Retell integration - On a Phone Caller agent, open Integrations, select Connect on the Retell card and enter the Retell API Key and Retell Agent ID, then select Sync from Retell on the Prompt page to import the prompt and tool catalog. The integration is optional, one provider can be active per agent, and TestMu AI never uses the credentials to modify your agent.
• Choose the tools each scenario must call - In the scenario editor, select the tools a scenario expects, such as check_availability and book_appointment for a booking. After each call, Expected Tool Validation marks every selected tool as Called or Not Called, and AUT Tool Calls shows everything the agent invoked, unexpected tools included. A Called verdict confirms the invocation only, so check in your own system that the booking exists.
• Give the test a number to call - For an agent that answers calls, register its number under Phone Numbers, with country code selection across 20+ countries, and TestMu AI dials that number with a real call. For an agent that places calls, the platform reserves a recipient number from an outbound pool for your Retell agent to dial, and because a Retell-bought from_number dials only US destinations, check which country the reserved number is in.
• Generate scenarios - Ask for up to 20 inbound or 7 outbound scenarios per generation from the prompt, the Conversation Flow or supporting docs, or write your own. Each scenario takes a persona, built in or custom, plus a language, special instructions and validation criteria, and each criterion is graded Pass, Fail or Unable to Verify and comes with its evidence.
• Set each caller's voice and line - Per phone scenario, pick one of 200+ voices with a choice of accents and speech speeds, one of 15 background noise presets such as street, call center or a poor cellular connection, a response timing between 0.5 and 5.0 seconds, and a maximum call duration between 60 and 1,800 seconds. You also choose who speaks first, and on outbound calls the agent does by default.
• Build a suite and run it - A suite groups scenarios with their voice and phone settings, test profiles that carry caller data a custom metric can reference as {caller_name}, and agent profiles, which are reusable caller identities with a name, number, voice and background noise. Run it from the dashboard or the CLI and follow each call's status and duration live, ending a call early if needed.
• Read the results - Every call is scored on 30+ metrics in 8 categories: conversation flow and interaction dynamics, accuracy and effectiveness, user experience and satisfaction, business operational metrics, audio voice quality, speech-to-text evaluation, validation results and automated issue tags such as latency issues, hallucination in call flow, patchy audio and running in a loop. Custom metrics grade your own rules from the transcript, each answered Yes / No or scored on a 0 to 100 scale that passes at 70 by default, and stay report-only until you switch them on in Metric Config, as the phone agent testing docs explain.
• Check the go-live verdict - The go-live assessment turns the run into a verdict, Green at a score of 80 or more, Yellow from 65 to 79 and Red under 65, and backs it with dimension scores for Functional Completeness, Quality Standards, Risk Profile and Operational Readiness, a confidence level based on call volume and prioritized action items.
• Load-test an inbound agent before a launch or a busy period - A performance test ramps calls at 5 per second to a target, holds it for 1 to 120 minutes (5 by default) and reports how much of the target load it held, how many calls connected and ended normally, and response latency from P50 to P99, without AI scoring of the conversations. It applies to agents that answer calls, its target cannot exceed your organization's parallels, and its calls count against the Retell workspace's concurrency quota, so a target above that quota also exercises Retell's wait of about 40 seconds and the fallback number.
• Score production calls - After launch, upload production call recordings in MP3 or WAV format to get the same metrics as test calls, and for an outbound agent, switch on passive mode, which listens to a live call and scores it without joining the conversation.

To test the speech pipeline without the phone network, the Voice surface reaches a Retell agent over Retell's web call API and scores it on the 9 quality metrics used for chat agents, as the voice agent testing docs describe. A Retell chat agent on the Chat API can share a regression plan with the phone agent, since Agent Testing covers chat and voice agents as well as phone, video and image agents. A single no-code session is walked through in the guide to test a Retell agent without code.

How Do You Keep a Retell AI Agent Reliable After Launch?

A Retell agent can start answering differently after go-live without any edit to its flow, for example when a knowledge base changes or an upstream model is updated, so a suite that passed before launch has to keep running.

• Treat every prompt edit, LLM swap, new voice, interruption sensitivity tweak, knowledge base update and new flow node as a release. Any of them can change how the agent handles a call while Retell's call logs look normal, so rerun the Retell batch and the phone suite after each one.
• Put the suite on a schedule with the Hourly, Daily, Weekdays, Weekly or Monthly presets or a custom cron expression, in the IANA time zone you choose, so an upstream model update that lands between your releases still gets caught.
• Add each failure a real caller hits to the suite as a permanent scenario, so over time the suite holds every case the agent has already failed once.
• Run the calls in parallel so the suite finishes fast enough to gate a merge, which keeps voice regressions inside the release process instead of incident reviews.

To gate a pipeline, install the CLI with pip install agent-testing-cli (Python 3.10 or newer), hold LT_USERNAME and LT_ACCESS_KEY as CI secrets and run the phone suite, as the Agent Testing CLI docs show:

# CI step: run a Phone Caller suite and wait for it
agent-testing-cli --project "$PROJECT_ID" run \
  --suite "$SUITE_ID" \
  --yes \
  --wait \
  --poll 5 \
  --timeout 1800

A failed test in a completed run returns exit code 1, a polling timeout returns 5, and either code fails the job. A Retell chat agent runs as a chat evaluation instead, which takes a workflow ID as well and runs without the wait flag.

Record which agent version every run tested. Retell's agent version docs describe environment tags that point to a version, so the production number can stay on a prod tag at V2 while you test V3, and Retell's batch history lists one version at a time; for a single API test call, override_agent_version does the same job.

After any change on Retell, run Sync from Retell again so scenarios are written against the deployed version of the agent, and fix any scenario the editor flags because one of its selected tools has left the catalog. Retell's deprecation notice adds that a dashboard conversion of a multi-prompt agent creates a new agent with its own ID, so point the integration's Retell Agent ID and your phone numbers at it and rerun the whole suite before it takes calls.

AI voice agent regression testing covers how to size and schedule a suite like this on any voice platform.

Conclusion

To test a Retell AI agent end to end, start by finding its type in the build-choice table and turning each failure mode above into a Retell simulation case or a phone scenario. Then connect the agent to TestMu AI's Agent Testing to run the phone scenarios on real calls, using the getting-started steps to test your first AI agent.