Barge-in Testing for Voice Agents: When Callers Talk Over

Barge-in as a Configured Behaviour

W3C VoiceXML 2.0, a W3C Recommendation dated 16 March 2004, describes barge-in in Section 4.1.5 as a user interrupting a prompt using speech or DTMF input. The spec treats it as an authoring choice, and only where the implementation platform supports bargein at all.

That choice lives in a bargein attribute on each prompt, honored only while that prompt is playing, and defaulting to a bargein property whose own default is “true”.

Section 4.1.5.1 defines the bargeintype values.

• speech - the prompt is stopped as soon as speech or DTMF input is detected, irrespective of whether that input matches a grammar and irrespective of which grammars are active.
• hotword - the prompt is not stopped until a complete match of an active grammar is detected, and input that does not match is ignored, including during the timeout period.
• No nomatch under hotword - because non-matching input is ignored, a nomatch event is never generated in the case of hotword bargein.
• A suggestion to the platform - the attribute suggests the behavior, and an implementation claiming bargein support is required to support at least one of the two types, not necessarily both.

Each type changes which events a suite can assert on.

• Under speech, a match is not required - detection alone stops the prompt, and the spec’s stated expectation is that the prompt stops as soon as the platform can reliably determine that the input is speech. Your result then rests on the platform’s speech determination rather than on any grammar.
• Under hotword, a valid caller can be ignored - someone who says something sensible that falls outside the active grammar gets no effect on playback, and no nomatch fires to record the attempt.
• The silent case needs an explicit assertion - a hotword miss shows up as a prompt that played to completion, so the assertion has to be on playback state and on the caller’s audio, since no error event arrives.
• Queued prompts drop after an interruption - the spec states that if bargein occurs during any prompt in a sequence, all subsequent prompts are not played, including those whose bargein attribute is set to false. An assertion that covers only the interrupted prompt misses the content the caller never heard.

On a VoiceXML-derived stack, a suite should record what was in effect for the utterance under test before its first assertion.

• Mode - speech or hotword as the platform applied it, since the document only suggests the type.
• Per-prompt value - the bargein value on the exact prompt that was playing when the caller spoke, because the attribute is honored only while that prompt plays.
• Platform capability - whether the engine supports the type you asked for, given that supporting both is not required of it.

A result logged without the mode, the per-prompt value and the platform capability is not reproducible, because the same audio replayed against a stack with a different bargeintype is a different test. I keep the mode in the test identity alongside the audio file and the grammar version. I scope the suite to a caller speaking over the agent mid-utterance, and DTMF barge-in and session-level interruptions carry their own cases.

Testing That the Agent Actually Stops

Barge-in here means the caller talking over the agent mid-utterance, a narrower case than the session-level breaks covered in voice agent interruption testing. RFC 6787, which defines MRCPv2, makes the stop normative for the synthesizer resource: with a SPEAK request active and kill-on-barge-in enabled, and the BARGE-IN-OCCURRED event received, the synthesizer MUST immediately stop streaming out audio. It MUST also terminate any speech requests queued behind the current active one, irrespective of whether those have barge-in enabled.

MRCPv2 splits detection from signaling across separate resources.

• START-OF-INPUT - an event from the server to the client indicating that the recognizer resource has detected speech or a DTMF digit in the media stream.
• BARGE-IN-OCCURRED - a method the client sends to the synthesizer resource, which RFC 6787 requires whenever the client receives a barge-in-able event from any source.
• Different resources - the recognizer reports, the client decides, the synthesizer stops. A server hosting both MAY optimize internally and stop audio without waiting for the client’s BARGE-IN-OCCURRED.
• Scope of the MUST - it binds MRCPv2 synthesizer resources and carries no authority over other protocols, so treat it as the model for your assertion and check separately what your own stack promises.

Detection failures and signaling failures leave different evidence in a run.

• Detection side - no START-OF-INPUT arrives, or it arrives late. The evidence is that event timestamp against the sample where the caller’s speech begins in the audio you injected.
• Signaling side - START-OF-INPUT arrived on time and audio kept playing anyway. The evidence is whether BARGE-IN-OCCURRED was sent, when, and what the synthesizer did after it landed.
• Why the split matters - a test that listens only to the output channel reports both as the same failure. Capture protocol traces alongside the recorded media so a failing run names which half failed.

Every assertion below reads off one recorded run.

• Audio stopped - no synthesizer frames reach the caller after the stop point. Check that on the recorded media. A dialog state machine only records that it issued a stop.
• It stopped soon enough - measure from the first sample of caller speech to the last synthesizer frame on the outbound stream, and hold that delta to a threshold you have agreed. Log it on every run so a rising number shows up before the threshold is crossed.
• Queued prompts did not continue - the recorded media must stay silent through the rest of the sequence, so extend the frame check past the interrupted prompt to everything queued behind it.
• The dialog layer agrees - VoiceXML 2.0 says a prompt’s bargein attribute is honored during the period in which that prompt is playing, and if barge-in occurs during any prompt in a sequence, all subsequent prompts are not played, including those whose bargein attribute is set to false.

Both rules are conditional. MRCPv2’s stop applies where kill-on-barge-in is enabled, which defaults to true when unset, and VoiceXML 2.0 treats barge-in as platform-dependent and as a per-prompt authoring choice. Where they apply, a queue that keeps draining after the caller interrupts is a defect, so assert against the tail of the sequence as well as the prompt that was playing.

The Fate of the Interrupting Words

Most barge-in assertions stop at the audio layer, and barge-in here means strictly the caller speaking over the agent mid-utterance. The suite checks that the agent stopped talking inside some window, then checks that its next turn was sensible. The recognized transcript for that turn goes unread, so the words spoken over the prompt are never checked against it.

VoiceXML 2.0 sets the behavior at the platform level. Section 4.1.5 specifies that when a prompt’s bargein attribute is false, input is not buffered while the prompt is playing, and any DTMF input buffered in a transition state is deleted from the buffer.

• Never collected in the first place - Section 4.1.8 restates it as input simply not being collected, which places the loss at the collection stage, before anything exists to discard; the explicit deletion clause covers keypresses already buffered, and speech over the prompt falls under the non-buffering rule.
• Gone by the time the prompt ends - under that attribute, what arrives during the prompt does not wait in line to be processed once the prompt finishes.

Your stack is almost not VoiceXML, but the shape of the failure carries into any implementation that suppresses input while a prompt plays. I have seen suites pass on stopped-audio timing alone while the agent was swallowing the opening answer of the call.

• The caller cannot tell the difference - someone who speaks during a prompt they are not allowed to interrupt has said something the system never received, and the call proceeds exactly as it would have if the words had arrived and been discarded.
• The next turn inherits it - they repeat themselves, raise their voice or change wording, and every downstream assertion now runs against a conversation the caller has already started correcting.
• The assertion belongs on the transcript - each barge-in case needs a check that the interrupting words are present in the recognized text, attributed to the right turn, with the right boundaries.

Each of the following needs its own transcript assertion, because the flow can advance correctly while the words are missing.

• The early answer - the caller gives the order number while the prompt is still asking for it. Assert the answer reaches the transcript and fills the slot, and that the next prompt moves on to the following field.
• The single word - the caller says “yes” across the prompt audio and nothing more. Assert that token is transcribed and consumed as the confirmation, because a silent drop and a clean confirmation leave the same trace when no assertion looks at the recognized text.
• The split utterance - the caller starts mid-prompt and keeps going after it ends, so one sentence lands on both sides of the boundary. Assert it arrives as one utterance whose first token matches the first word the caller spoke, timestamped before the prompt ended.
• Log the retry as a result - Section 4.1.5.1 names the caller-side symptom as the “stutter” effect, a caller who stops mid-utterance and starts again if nothing signals that the first attempt was received. The spec scopes that to the speech bargeintype, where detecting input stops the prompt without waiting for a complete grammar match, and frames it as a design expectation with no MUST-level force.

False Triggers From the Agent's Own Audio

The wider audio-quality picture is covered in voice quality testing. An agent utterance stops partway through, the transcript shows no caller turn at that moment, and the scenario passes on the next run.

MRCPv2 defines endpointing as “the process of automatically detecting the beginning and end of speech in an audio stream”, and its START-OF-INPUT event reports only that the recognizer resource “has detected speech or a DTMF digit in the media stream”. Neither wording identifies the speaker.

When the agent’s own outbound audio finds a return path into the inbound channel, the protocol chain runs:

• The detector fires - inbound audio carrying the agent’s own playback satisfies START-OF-INPUT, and the recognizer reports speech on the caller leg.
• The client signals barge-in - the MRCPv2 client sends BARGE-IN-OCCURRED to the synthesizer resource, though a server hosting both resources may stop the audio internally instead.
• Playback is killed - with kill-on-barge-in enabled, the default when the header is unset, the MRCPv2 synthesizer resource MUST immediately stop streaming audio.
• Queued speech dies with it - the same event terminates the speech requests queued behind the active one, irrespective of whether those have barge-in enabled.

The variables that decide whether leakage happens sit in the audio path, out of reach of the model and its turn-taking prompt:

• Endpoint coupling - speakerphones and open-air handsets put agent playback back into the microphone.
• Echo control state - cancellers can take time to converge, and reconverge after a path change.
• Mixing in the route - a bridge or gateway that mixes directions can present outbound content on the inbound leg.
• Codec and transcoding - re-encoding reshapes whatever residual reaches the detector.

Because those variables change per call, a scenario that passes over one device and codec pair can fail over the next route your traffic picks, which is why the run history reads as intermittent.

A defect that only turns up when the route hands it to you is hard to work on, because you cannot tell a real fix from a call that happened not to leak. TestMu AI’s agent testing platform carries an echo and reverb condition in its acoustic presets, and background noise is a multi-select axis in matrix testing, so an echo condition becomes an input you set on the run. Which condition your own callers meet on a given route is still something you work out from your own traffic, but a condition you can impose is one you can re-run a scenario against after a change.

What to look for in the evidence:

• Separate channel captures - record inbound and outbound to distinct files, because a single mixed recording cannot separate the two legs.
• The stop with nothing behind it - a stop event whose timestamp lines up with silence on the caller channel is the tell.
• Collapse of what was queued - on a platform that supports bargein, VoiceXML 2.0 honors each prompt’s bargein attribute only while that prompt plays, and once bargein occurs all subsequent prompts are not played, including those set to “false”.
• Path metadata on the run record - device, codec and route stored beside the result, so the correlation is findable later.

The test that isolates it keeps the caller silent:

• Pick a long utterance - a confirmation readback or a disclosure gives the leak time to build.
• Inject nothing - the caller leg carries no speech and no DTMF for the whole duration.
• Assert completion first - check that the utterance reaches its end; the absence of a BARGE-IN-OCCURRED is weaker evidence, since a co-hosted server may never send one.
• Repeat per path and within each path - one pass over one route proves little about an intermittent leak.

A silent caller looks like a scenario with nothing in it, so the case is easy to leave out. Run it on every route that carries production traffic. It stays inside the barge-in boundary, where the interruption under study is the caller speaking over the agent.

The Endpointing Trade-off

Scoring a full call end to end is covered in testing AI calling agents. RFC 6787, the MRCPv2 specification, defines endpointing in its definitions section, Section 2.1, as “the process of automatically detecting the beginning and end of speech in an audio stream.” One detector marks both edges: the beginning edge is what stops the agent mid-utterance, and the end edge is what decides the caller has finished. The RFC defines the term; where the end-of-speech threshold sits is an implementation setting, a silence duration governing how long the detector waits after speech stops before declaring the turn over.

• Short threshold - the agent is quick to treat a pause as the end of a turn. A caller who hesitates mid-sentence, reads a long reference number in chunks, or pauses to check a card gets cut off before finishing.
• Long threshold - the agent waits out the pause before responding. The caller finishes speaking and hears nothing back, which reads as a slow or unresponsive agent.
• Tuning one side moves the failure to the other - a shorter threshold clears the dead air and puts cut-offs in its place, so complaint counts from a single category will not tell you where the setting belongs.

Endpointing settings live in the recognizer configuration, where they can change between runs without appearing in any test artifact.

• Record the endpointing configuration with the run - alongside the model version, the prompt version and the audio path, as a stored field on the run record, so later comparisons can filter on it.
• Two runs at different settings are not comparable - the pass rates were produced under different conditions, so the delta between them measures the configuration change as much as the agent.
• A configuration change can present as a regression - if the threshold was moved to chase a latency complaint, the next barge-in run shows a step change that has nothing to do with the agent.
• Sweep the threshold in one session - when you change the setting, rerun the same scenarios at the old and new values back to back so both sides carry recorded conditions.

Under the speech bargeintype, VoiceXML 2.0 says in Section 4.1.5.1 that stopping the prompt as early as possible is “desireable” [sic] because it avoids the “stutter” effect, in which a user stops in mid-utterance and re-starts if he does not believe that the system has heard him. The specification words this as a design expectation and scopes it to the speech bargeintype; hotword holds the prompt until a complete match of an active grammar is detected, so the expectation does not carry over.

• Stutter is visible in the caller audio - a truncated word, a gap, then the same phrase started over. Score it separately from the agent-side timings, since the agent may look correct while the caller is repeating themselves.
• Stutter sits on the beginning edge - the prompt is still playing after the caller has started talking, so nothing has acknowledged the interruption and the caller re-starts.
• Do not write it as a conformance failure - the language it comes from is an expectation about good design under one bargeintype, so a stutter check belongs in the suite under its own name and standards-compliance wording stays out of the report.

Scenarios Worth Writing

The thing under test is a detector deciding when speech has begun, so a harness feeding the agent typed turns skips the behaviour. The fixture has to land the same overlap on every rerun.

Landing the same overlap on every rerun settles the timing, and how readily a caller cuts in is a second variable that travels with the caller from scenario to scenario. Write that disposition into each case and it drifts, so the pushy caller in your mid-word scenario and the pushy caller in your queued-prompt scenario become separate definitions you have to keep in step. A harness that carries that tendency on a reusable caller persona, applied across runs and suites, keeps the interrupting behaviour in one place and lets you put several of those personas on one run, where each variant’s scenarios run in parallel.

• Audio-stream fixtures - MRCPv2 defines endpointing as automatically detecting the beginning and end of speech in an audio stream, and a text fixture supplies no stream.
• Controlled overlap onset - the fixture sets the millisecond at which caller audio starts relative to the prompt, so an interrupt placed mid-word lands mid-word on every rerun.
• Per-channel recording - caller and agent captured as separate tracks on a shared clock, stored with the run’s event log.

Summed into one track, an overlap region is only louder audio. A late agent stop and an early caller start give the same waveform, so neither edge lands on the timeline.

Barge-in proper covers the caller talking over the agent mid-utterance; dropped calls, holds and transfers belong to their own suite. Each setting below changes the expected result for the same scenario.

• Per bargeintype - VoiceXML 2.0 defines speech, where the prompt stops as soon as speech or DTMF is detected irrespective of grammar match, and hotword, where it does not. The attribute suggests the behaviour to the platform without guaranteeing it, and a platform claiming bargein support implements at least one of the two types.
• Per prompt setting - in VoiceXML 2.0, with bargein false, input is not buffered while the prompt plays and DTMF buffered in a transition state is deleted, so the passing result is lost input.
• Per kill-on-barge-in state - MRCPv2 defaults it to true when unset, and correct behaviour flips with it, so the fixture pins the value and the run log carries it.

Treat the bargeintype value as a configuration axis of the run, and pair every stored artifact with the value active at capture. A pass recorded under one value says nothing about the other.

Conclusion

Record the configuration before you write a single assertion. Bargein mode and endpointing settings decide what correct behaviour even is, so a result without them is an anecdote rather than a measurement.

Then add the assertion almost nobody has, which is what happened to the words the caller spoke during the overlap. The agent stopping is the visible half, and the lost utterance is the half your callers actually complain about.

Write the silent-caller test last and keep it forever. It costs one scenario, it catches a class of defect that otherwise reads as flakiness, and it is the only test here that fails when nothing happened at all.