Independent Verification and Validation (IV&V): What It Is and How to Apply It to AI Agents

What Is Independent Verification and Validation (IV&V)?

Independent Verification and Validation (IV&V) is the practice of having a separate organization check that the software has been built correctly and that it is the right software. The IEEE 1012 standard defines IV&V as verification and validation "performed by an organization that is technically, managerially, and financially independent" of the development organization.

The two halves answer two different questions. NASA's IV&V Program puts it this way:

• Verification asks "Are we building the product right?" It's a check that what you've produced in each phase actually meets the requirements for that phase.
• Validation is all about 'Are we building the right product?' It's about making sure the product is in fact meeting the mission and the customer's needs.

IV&V puts the word 'independent' in front of both. The independent team decides what to test, how to test it, and when, and then reports what it finds without the builder having any opportunity to edit the result.

What Are the Three Types of Independence in an IV&V?

IEEE defines independence along three parameters. NASA describes each one:

Type of independence	What NASA describes	What it means for AI agents
Technical	IV&V staff assesses development processes and products with their own expertise, independent of the developer.	Test scenarios, simulated callers, and graders come from a different system than the one that runs the agent.
Managerial	Responsibility for IV&V sits in an organization separate from the team that builds the system; the IV&V team chooses what to analyze, which techniques, the schedule and which issues to pursue.	The team that signs off on release is not the team that tuned the prompt, and the agent's platform does not decide what "pass" means.
Financial	The IV&V budget sits with an organization independent from development.	You do not depend only on the vendor whose agent is under test to tell you whether it is ready.

Why Is IV&V Relevant for AI Voice Agents and Chatbots at This Point in Time?

AI agent platforms now ship their own testing. That's good news for the builders, and it does raise a fairly legitimate question: if the platform that runs your agent also plays the caller, scores the call, and reports the score, who actually checks the platform?

Here is what four major platforms deliver as of 14 November 2026:

• Vapi offers Evals (text-based mock conversations which are checked by exact match, regex, an AI judge or tool call checks) and Simulations (an AI tester acts as the caller in chat or voice mode and is scored against success criteria). Vapi's own documentation states that Evals "do not test the voice, transcriber, audio quality, or turn-taking behavior". (Vapi Evals, Vapi Simulations)
• Retell AI offers simulation testing (scenarios saved as graded test cases), batch testing after every change, and Conductor, which will draft test cases, run batches, and report failures. You can hear the real audio via web and phone calls, but the Retell docs state "Only simulation testing grades a run" and "custom LLM agents can only be tested with a web or phone call." (Retell testing overview)
• NiCE Cognigy launched the Cognigy Simulator in January 2026, described as an 'AI performance lab' which runs synthetic customers against an agent and scores on task completion, guardrail adherence, integration reliability and experience quality. The Cognigy docs state that "The Simulator only supports text-based conversations". (NiCE press release, Cognigy docs)
• Genesys Cloud enables teams to test agentic virtual agents via the AI Studio preview widget, the Architect bot flow simulator for digital bot flows, and flow execution history with replay mode. (Genesys Cloud help)

Each of those tools is useful as you're building. None of them is in any way independent of the platform under test, and none of them purports to be.

That's precisely where IV&V fills the gap. It's a check that doesn't share the builder's runtime, simulator, or grader.

There's a second reason. Many organizations run multiple agent platforms. A contact center might have IVR on Genesys, a chat agent on Cognigy, and an outbound voice agent on Vapi or Retell.

You get four built-in test tools, which give you four different scoring systems that are very difficult to compare. An independent test layer can take the same set of requirements and run them against all of them.

How Do the Built-In Platform Testing Tools Stack Up?

This table sets out what each platform's own documentation states (checked 14th November 2026). "Not stated" means it's not mentioned on the public page, not that the feature is absent.

When Is a Platform's Own Built-In Testing Sufficient?

Built-in testing is the right tool for a lot of the work. Use it and add independent testing when you need to.

Built-in testing is usually enough when:

• You're prototyping one agent on one platform, and you just want some fast feedback as you're playing with the prompts.
• You need to check the exact tool call and its arguments at that one point. Vapi Evals are for that.
• You want to test how your Cognigy agent deals with third-party API failures. NiCE states that the Cognigy Simulator will emulate "the full range of third-party API responses, from clean paths to rare error conditions".
• The agent is internal, low risk, and not yet talking to any customers.

Add independent verification and validation when:

• The agent talks to the customer on the phone, and you need proof of how it actually sounds on the phone: the latency, the interruptions, the accents, the background noise, the keypad (DTMF) input and the transfers.
• You run agents across more than one platform and require a single scorecard.
• A release needs sign-off from someone other than the team that built it (QA, compliance, risk or a client).
• You switch models/prompt/platform and need the same tests before and after to see what's changed.
• You need evidence that a reviewer can open months down the line without having to log into the agent vendor's console.

What Does Independent Testing Add on Top of the Built-In Tools?

Independent testing for AI agents delivers five things that a platform's own toolset isn't set up to provide.

• A separate runtime. The test system makes the call/chat to the agent in the same way a customer would, from outside the platform. A built-in simulator that talks to the agent from inside the platform can bypass the phone number/carrier/routing layer where so many of the real call failures occur.
• One test suite across platforms. The same scenarios, personas and pass/fail rules can be run against a Vapi agent, a Retell agent, a Cognigy agent and a Genesys flow so the results are comparable.
• Real channels. Real phone calls through a phone number or SIP endpoint. Real audio. Real DTMF tones. Text-only simulators cannot tell you anything about how speech recognition will handle a caller on a noisy street.
• Requirements you own. Test specs are taken from your PRDs, your tickets and your policies, not from the agent config. That's the essence of technical independence: the test doesn't inherit the builder's assumptions.
• Evidence and a release gate. Each run concludes with recordings, transcripts and a pass/fail result, which can in turn block a deploy in CI (just as a failing unit test would block a code merge).

Isn't Every Testing Vendor "Independent"? How to Judge IV&V Providers

Being a third party is the starting point rather than the end point. A number of test vendors have already made the case.

Klearcom talks about "Independent voice assurance" and points out that an AI provider testing its own agent is "marking its own homework". Cyara refers to its new AI Trust Observability as a "vendor-neutral layer". Cekura states that you can "run the same scenarios across platforms and models".

So "we're neutral" does not separate one IV&V provider from another. These questions do:

How Do You Do IV&V on an AI Voice Agent, a Chatbot or an IVR?

Independent verification and validation of an AI agent is exactly the same as IV&V of any other software: separate requirements, separate execution, separate verdict. The seven steps work regardless of whether the agent is running on Vapi, Retell, Cognigy, Genesys or a custom stack.

• What "correct" means is to write it down outside the platform. Get the PRD, the call scripts, the policy docs, the escalation rules, and the Jira tickets for the release. That's the source of truth. Not the agent's prompt.
• Turn requirements into fixed test scenarios. Each scenario should name the caller's goal, the persona, and the expected outcome (e.g., "caller asks to reschedule a delivery; agent confirms the new date and sends a confirmation"). Keep the scenarios and pass/fail rules exactly the same from release to release.
• Set your validation criteria before you do anything. Work out the rules up front: what disclosures are required, when you hand off to a human, what's the maximum latency, which routes the IVR menu must get to. That's how bias creeps in if you're deciding your pass criteria after you've seen the results.
• Get to the agent the way the customer does. Make real calls to the number or SIP endpoint, send real chat messages, press the real keypad tones. Vary the accents and background noise.
• Run the same suite on every platform that you use. If you're comparing Vapi against Retell, or moving an IVR from one CCaaS to another, the suite should be exactly the same.
• Keep the evidence. Keep the recording, the transcript with the timestamps, and the verdict for each scenario. Someone outside the build team should be able to go and look at a failure without having access to the agent vendor's console.
• Make the verdict a release gate. Run the suite in CI on every prompt, model, or flow change and block the release if it fails. Post-launch, do an analysis of the production recordings against the same criteria.

How Does TestMu AI's Agent Testing Support Independent Verification and Validation?

TestMu AI's Agent Testing tests chat, voice, phone, IVR and video agents that are running on a platform outside of TestMu. It can therefore act as an independent check. The feature names below are as they appear on the TestMu AI product pages.

• Scenarios from your requirements. Agent Testing automatically generates 60-100+ test scenarios from your uploaded PRDs, PDFs, DOCXs, images, audio and video, as well as from connected Jira and Confluence. Test Profiles and Personas contain reusable test data.
• Fixed pass/fail rules. Validation Criteria define the rules that each conversation is assessed against, with configurable pass/fail thresholds so the same criteria can be applied to every release.
• Real calls, real conditions. Live test calls will simulate inbound and outbound traffic with 200+ voice profiles covering 50+ accents and dialects, 15 background noise presets and DTMF keypad detection. IVR tests will check menus, routing, queues, and transfers and will return Pass, Fail or Unable to Verify for each expected route.
• Metrics for the words and the call. Chat and voice agents are scored on 9 Quality Metrics; phone agents on 30+ Call Metrics (e.g. First Call Resolution (FCR), intent recognition, containment rate, voice quality, speech-to-text accuracy).
• Evidence for every session. A downloadable recording and a turn-labeled transcript with timestamped evidence.
• A release gate. The Go-Live Assessment produces a Green, Yellow, or Red verdict. Continuous agent testing in CI compares each run against the previous one, and runs can be scheduled using cron expressions.
• Cross-platform coverage. TestMu AI's agent platforms page lists 25 voice and IVR platforms and 22 chat platforms. These include Vapi, Retell AI, Cognigy (chat scoring and live calls into a Cognigy Voice Gateway SIP endpoint), and Genesys Cloud CX.
• After launch, Production Recording Analysis will check the uploaded call recordings against the same criteria.
• One vendor for the rest of the stack. The same company provides the web and mobile app testing via KaneAI and the Real Device Cloud, so you don't need separate vendors for agent testing and app testing.

For channel details, see voice agent testing and IVR testing with real phone calls.

Conclusion

Pick one customer-facing agent, usually the one on your main phone line, and write its pass/fail rules from your PRD, policies and tickets before the first run. Keep the platform's built-in tests for fast feedback while you build, and run one independent suite against the deployed agent before each release.

The guide to testing your first AI agent shows how to connect an agent to Agent Testing.

Testing an AI agent? TestMu AI's Agent Testing validates chat, voice and IVR agents before your customers talk to them.

To run your first test, start free, or

book a demo

for a full walkthrough of the product.