Testing OpenAI Realtime Agents: A Practical Guide

What Is an OpenAI Realtime Agent?

An OpenAI Realtime agent is a voice agent built on the Realtime API, where a single speech-to-speech model takes audio in and produces audio out. OpenAI contrasts this with traditional pipelines that chain multiple models across speech-to-text and text-to-speech, and the stated reason is latency and nuance rather than convenience.

The architecture is the part that changes your test plan. A chained pipeline hands you a transcript between stages, and most voice test suites were built to assert on exactly that string. One model doing both ends removes it.

The API reached general availability in August 2025 alongside the gpt-realtime model, and the release added capabilities that widen the surface you have to cover.

• Remote MCP servers - passing a server URL into the session configuration makes those tools available, and the API handles the calls, so your agent’s tool surface can change without a code deploy.
• Image input - images can join an audio session, which means a conversation can be grounded in a screenshot the user is looking at.
• SIP phone calling - the agent can answer the public phone network, PBX systems and desk phones directly.
• Asynchronous function calling - long-running calls no longer stall the session, so the model keeps talking while a tool result is pending.

Each of those is a test surface. An agent that reads screenshots can be fed a misleading one, and an agent that talks while waiting on a tool can answer from stale context. For the broader category this sits in, our guide to conversational AI testing covers chatbots and voice agents together.

OpenAI’s Published Benchmark Scores

The strongest argument for testing a Realtime agent comes from the vendor. In its gpt-realtime announcement, OpenAI published three audio benchmark scores alongside the numbers its previous December 2024 model reached.

Read the third row against the first. The model that reasons over audio best is the same one that holds a multi-turn instruction worst, and instruction following is what your system prompt is made of.

Every improvement here is real and every score is the vendor’s own best case, measured on its own evaluations. A production agent carries your prompt, your tools and your edge cases on top.

Assertions for a Realtime Agent Test

With no intermediate transcript to trust, assertions move onto what the session observably did. Each row below is a separate check, and each fails on its own.

The two interruption rows are the ones teams collapse into one. Cancelling generation server-side and stopping playback client-side are different code paths, and a suite that only checks the first will pass an agent that talks over every caller.

For how these checks differ from judging a reply on quality alone, AI agent testing methodology compares manual review, LLM-as-a-judge and simulation.

Transport Choices and What They Change

A Realtime agent can be reached three ways, and the transport decides which failures your suite is even capable of producing.

Phone is the one that punishes a browser-only suite. Audio that survives a clean WebRTC session can lose enough fidelity through carrier compression that the alphanumeric readback assertion starts failing, and that is a genuine production defect rather than a flaky test.

Test the transport you ship on. If the agent answers phones, a passing WebRTC suite is evidence about a surface your users never touch.

Note: Connect an OpenAI Realtime endpoint and run generated voice scenarios against it, scored on the same metrics every run. Try TestMu AI free!

Failure Modes Worth a Test Case

These come from what the Realtime API newly supports, which is where the untested surface tends to be.

• Stale answers during async tool calls - the model can keep the conversation fluid while waiting on a long-running call, so write a scenario where the tool returns late and assert the agent does not answer from what it assumed in the meantime.
• Tool surface changing underneath you - pointing a session at a different remote MCP server changes the available tools without a deploy, so a suite pinned to yesterday’s tool list stops testing the agent you are running.
• Misleading image context - an image joins the conversation as context the model grounds its answers in, which makes a wrong or stale screenshot an input worth a negative test.
• Mid-sentence language switching - the model can switch languages within a turn, so assert that it switches when the caller does and holds the line when they do not.
• Sessions halted by classifiers - OpenAI runs active classifiers over Realtime sessions and can halt conversations detected as violating its content guidelines, which your suite should recognise as a terminated session rather than an agent failure.

That last one matters for triage. A halted session and a failed assertion look similar in a summary and mean completely different things to the engineer reading it.

Teams evaluating what to run this on can compare options in our roundup of AI voice agent testing tools.

How to Gate a Realtime Agent in CI

Run the suite on every prompt change, not only on code changes. A system prompt edit is a behaviour change on the axis the benchmark table above scores lowest, which makes it exactly the change most likely to regress.

• Pin the scenarios to your declared tool surface so a tool-list change fails loudly rather than silently reducing coverage.
• Separate agent defects from transport errors in the result, so a dropped session never reads as a product bug.
• Run the phone surface on its own schedule if it is slower, but never let a WebRTC pass stand in for it.
• Keep the audio, not just the score, so a failed readback assertion can be listened to rather than argued about.

TestMu AI’s Agent Testing is built for this half of the problem. It lists REST, WebSocket and OpenAI Realtime among its voice endpoint transports, so a Realtime agent connects as a configured endpoint, and it scores chat and voice agents on 9 standardised metrics and phone calls on 30 or more, including DNSMOS P.835 audio quality scoring.

The output is a production-readiness verdict rather than a scoreboard: Green for ready, Yellow where targeted fixes are needed, Red for not production ready. Setup is covered in the Agent Testing documentation, and a no-code walkthrough of the same loop is in how to test a voice agent.

Conclusion

Take the single flow your Realtime agent handles most often and write two assertions against it: the tool call it must make, and the interruption it must stop for. Run both on the transport you actually ship on.

Those two cover the axes OpenAI’s own numbers say are weakest, and they fail loudly enough to be worth a pipeline gate. Connect the endpoint and read the first verdict before you tune anything else.