Testing the Agents That Look You in the Eye [Testμ 2026]

Agents With a Face

Srinivasan traced the arc that got the industry here. First came websites and mobile apps, where you click, find elements, and automate against deterministic behaviour, which open-source communities sustained for the best part of a decade.

Then came agents: chat agents, voice agents, phone callers, WhatsApp agents, and multi-agent systems with sub-agents underneath. TestMu AI had covered evaluating those at the previous year’s conference.

The video agent is the one that breaks the pattern. It does not only talk, because an avatar, its expressions, and the sentiment behind them are all part of what the user receives.

These agents are already customer-facing. They triage patients, onboard customers, and conduct interviews on camera, acting as the gateway to a business, which is what makes the failure modes worth taking seriously.

The Scale Problem

Sai worked through the arithmetic that makes manual testing untenable. Start with one scenario, which is already an understatement, since a real suite covers happy paths, negative paths, and red teaming.

That single scenario has to run against different personas, genders and avatars, and then repeat, because a conversational agent does not behave identically twice. Srinivasan later put the matrix for one scenario at roughly seven and a half hours.

Time is only the first constraint. The harder one is supply: even a team willing to spend the hours cannot produce 10 to 15 genuinely different personas, accents and dialects on demand from the testers it happens to have.

Automating the Human

Srinivasan’s answer to the supply problem was direct. If the only way to evaluate a conversation is to have one, then the human side of that conversation is what you automate.

That candidate carries personality, an avatar face, lip sync, expressions, and a language along with the dialects inside it. Geography and ethnicity matter too, because a video conversation carries all of them whether or not the test accounts for it.

They build it in three layers:

• The avatar - the face the agent under test actually sees on camera.
• The brain - the core that thinks, reasons, and acts on its own to build the next turn, interpreting facial expressions, lip sync and dialect as it goes.
• The browser - a real session joining the link, where the camera and mic are faked so a TestMu AI candidate stands in for the human.

Turn taking has to happen in near real time. As soon as the candidate hears something it has to perceive, understand and respond quickly enough that the exchange still sounds human.

Srinivasan credited more than a year of prior work on voice agents for making this feasible, since the models and architectures on the voice side had already matured. Video then adds facial expression, lip sync and background on top.

Memory and Red Teaming

Sai raised a scenario that only appears in multi-round interviews. A candidate clears round one, and a different avatar turns up for round two.

Catching that requires the brain to remember rather than decide, act and forget. Srinivasan confirmed memory holds not just what happened in the first round but the video evidence of who attended, under which persona, and what the evaluation concluded.

He tied it to a familiar hiring problem, where the person on screen one and the person on screen two for the same role were not the same person. Without carried-forward memory, an interview agent has no way to notice.

No SDK, No Test Hooks

Sai said the most common customer question is how their agent connects to the test system, and that the honest answer usually meets resistance. Teams do not want to add test hooks or change code purely to make evaluation possible.

Their approach avoids the question. Anything reachable through a joinable URL works, whether that is Zoom, Google Meet, WhatsApp calling, or a call hosted inside a product.

No SDK, no injected JavaScript, no modules, and nothing to change on the customer side. The test candidate arrives the way any other participant would, as a third party joining the call.

Live Demo

Sai wanted to play a recording, Srinivasan wanted to run it live, and they put it to the audience. The chat voted for live, so the demo ran live.

The platform is available through a user interface, a CLI, and MCP. The input is context about what the agent should and should not do, supplied as a system prompt plus video or audio recordings, since a prompt alone rarely carries enough.

From that context it generates scenarios across categories including happy paths and red teaming, with special instructions available to push at a particular case. Srinivasan stressed that this is heavy context engineering rather than uploading files to a model and asking for scenarios.

Each generated scenario carries a goal, characteristics such as who speaks first, and expected output in the form of validation criteria. Users can add their own criteria and their own personas, keeping a human in the loop over what counts as correct.

Those combine into a suite spanning personas, avatars, test data and a test profile, with each persona triggering its own sessions. The runs execute on HyperExecute, which already carries voice and chat agent scenarios.

The demo target was a video interviewer agent with a face and a voice, joined over WebRTC. While the live run executed, they walked through an earlier one recorded in August: two avatars interviewing each other, one asking about experience and strengths, the other answering as a backend engineer with five years of experience.

Afterwards the platform produces a full transcript and then the evaluation. It reports whether the goal was reached, checks each validation criterion with supporting evidence, and scores conversational flow, question handling, response quality and avatar presentation.

Crucially the evaluation is multi-turn rather than single-turn, so it can tell whether the conversation held together from beginning to end or drifted somewhere in the middle.

The Hardest Problems

Asked what actually took the longest, Sai named lip sync first. Getting the fine-tuning right so the two agents exchange turns smoothly, without visible delay, was the toughest single piece.

The subtler problem was audio. Root mean square measures how much sound sits in a clip, where zero means silence and ordinary speech runs into the thousands, which in decibel terms puts human conversation roughly in the 15 to 34 range.

They spent a long time debugging readings around 0.2, assuming such a low value meant silence. It did not, and something was happening underneath it.

That produced a rule worth carrying into any conversational testing: silence is not automatically a failure. An interview agent asking a question and meeting a pause has not necessarily broken.

Srinivasan added that the fault could sit anywhere along the chain. The browser might not be rendering, the browser might not be decoding the audio, or the agent under test might simply not be responding.

Turn Taking

The third hard problem is one everybody has experienced on a bad video call. Sai broke it into three states the test candidate has to be able to produce and the agent has to handle.

• Too eager - the candidate talks over the interviewer, creating overlap and interruption, which is how an anxious or interrupting persona behaves.
• Too patient - the candidate leaves dead air, and the interviewing agent fills it with reassurance about taking your time.
• Balanced - the exchange lands where a human conversation would, which is the behaviour they had to teach explicitly.

Watching one machine politely tell another to relax is a good illustration of why this needs testing at all. Tuning their own candidate to produce these states was what let them check whether the agent under test copes with them.

Sai’s summary was that silence is not the end of a turn but a comma inside one, and where it falls varies from persona to persona.

Metrics and What’s Next

When they launched voice agent evaluation a year earlier it carried two or three metrics. It now runs to more than 15, including mean opinion score, which they added recently.

Video expands the set again, because avatar presentation matters: how the lip sync looks, what the facial expression conveys, and whether audio and video actually match. Alongside that sit conversation quality, question handling, conversation flow, and goal drift over a long exchange.

Srinivasan set a useful bar for observability. If you could not confirm something by watching the recording yourself, the grader cannot confirm it either.

Their practical advice was to decide what good looks like before the run rather than after, run it more than once, and account for the bad days: nervous people, rude people, silent people, and whether the agent showed empathy to any of them.

Sai was blunt that a single passing conversation proves nothing. They have watched the same scenario fall apart against a persona who answers only yes and no.

On what comes next, three things are in progress:

• Native apps - joining calls through installed Zoom, Teams, Google Meet and Webex clients rather than only through the browser.
• Noisy backgrounds - people moving through an office, crowded rooms and interruptions, instead of the plain background the demo used.
• Multi-person red teaming - two people in frame, to see whether an agent expecting one participant notices and stops.

Q & A Session

Mudit put audience questions to both speakers at the end.

• How does chaos engineering apply to the video modality?

  Srinivasan: Introduce noise and busy backgrounds with other people talking, and you are testing voice activity detection directly. The question becomes whether the agent picks up the person it is talking to or the conversation behind them. That pattern already exists for voice and phone caller agents, and it carries straight into video.

• How do you evaluate nodding and looking attentive?

  Sai: Those are metrics that separate video from phone callers, covering things like eye blinking and expressed sentiment on the call. Because evaluation runs against the entire video rather than a transcript, these can be scored against the same criteria, and he confirmed they are being brought in.

• What causes the latency in a live video agent?

  Srinivasan: Plenty of models generate video, but a live agent is a different problem, because the voice has to be generated in real time with no delay while expressions and lip sync stay matched to the sound. Sai added that they deliberately avoid the speech to text, then LLM, then text to speech route, since each hop adds delay.

• Can the agent handle sign language?

  Sai: No, and he was candid that they had not considered it. He thanked whoever asked, called it a feature worth adding, and treated the question as a genuine gap rather than talking around it.

• How does scenario generation change for a new modality?

  Srinivasan: Generation for a chat agent differs from a voice agent, which needs languages, dialects and geographies, and video adds ethnicity and accessibility on top. They bake in guardrails and run multiple grader agents behind the scenes so nothing drifts from the context the customer supplied, and video aesthetics such as blinking and nodding now feature both in generated scenarios and in validation criteria.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.