Voice Agent Latency Testing: Where the Clock Starts

Two Numbers, Same Agent

ITU-T G.114 (05/2003) states in clause 4 that although a few applications may be slightly affected by end-to-end delays of less than 150 ms, “if delays can be kept below this figure, most applications, both speech and non-speech, will experience essentially transparent interactivity.”

• 150 ms, mouth-to-ear - G.114 attaches this figure to end-to-end delay at the application level, so the window already holds every hop the caller sat through, endpointing included.
• 400 ms one-way, scoped UNI to UNI - the value G.114 recommends not exceeding for general network planning, so it answers a different question from the 150 ms figure.
• Both sit at guidance strength - compliance with the Recommendation is voluntary, and G.114 says nothing about voice agent stacks, so reading either as a yardstick for one is my application of it.

The component-level view of a voice pipeline is covered in voice quality testing. Put two teams on the same agent, the same build and the same scenario set, and the reported latencies can differ by enough to change whether the agent looks acceptable. The start event is what moves; the stop event usually sits at the first frame of agent audio.

• Caller stops speaking - the clock starts at the last audio frame that carried caller speech, so the window carries endpointing delay and everything downstream.
• Recogniser emits a final transcript - the clock starts when the transcript is committed, which drops the wait the caller sat through before that commit.
• Request reaches the model - the clock starts at the serving boundary, which is the slice that team owns and can change this week.

Mixed start points break every comparison the number gets used for.

• Against a guidance figure - G.114 states its 150 ms for end-to-end delay, mouth-to-ear in the case of speech, so a serving-boundary number gets read against that figure while excluding time the caller lived through.
• Against another team - two dashboards can report visibly different figures for identical caller experience, and the argument about which stack is slower never resolves, since neither side describes the same window.
• Against your own history - an instrumentation change moves the reported number without touching the agent, and the regression you chased last month turns out to be a start-point edit.

Clause 1 of the 2003 edition records that earlier versions carried a 150 ms application level figure alongside a 400 ms network planning limit, and that “this parallel treatment of network delays on one hand, with application (‘mouth-to-ear’) level delays on the other hand, led to confusion in how ITU-T Rec. G.114 should be applied.” ITU-T revised it to give E-model-based guidance instead.

I have sat through a latency review where the disagreement turned out to be two dashboard panels built off different timestamps that nobody had written down. Every latency figure I publish now carries its window definition in the panel description.

• Named start event - the specific signal, at the specific place in the pipeline, that stamps t0.
• Named stop event - the specific signal that stops the clock, stated with the same precision.
• Stated window contents - what the span includes and excludes, so a reader can tell whether your figure is comparable to theirs.
• Owning team - who can change the instrumentation, so a reader who sees the number move knows who to ask.

The Start and Stop Points

Reliability objectives built on these measurements are covered in AI reliability engineering. Every latency number you already hold encodes a start instant and a stop instant. Unless your test plan names those two instants, the choice was made by whichever library emitted the timestamp. Candidate start points on the same turn sit at different moments, and the distance between them is the wait your chart never records.

• The caller physically stops speaking - the acoustic end of their last word. The caller’s sense of waiting begins here, and it covers everything your stack does afterwards, whether or not you own that part.
• The system decides the caller has stopped - always later than the first, because it sits behind endpointing delay, which measuring from here hides entirely.
• A request leaves your service - the most convenient point to instrument and the narrowest. It excludes carrier transport, media ingestion, buffering and the endpointing decision, which scopes the number to one hop of the turn.

When latency reports look healthy and callers still complain about waiting, check the start point first. Measuring from the third one leaves carrier transport, media ingestion and the endpointing window out of the number.

ITU-T Rec. G.114 (05/2003) records in its own Introduction that carrying network delays alongside application level, or “mouth-to-ear”, delays “led to confusion in how ITU-T Rec. G.114 should be applied”. Its figures still travel with their reference points attached.

• 400 ms one-way, scoped UNI to UNI - the value the Recommendation advises not exceeding for general network planning. Quoted without that scope, it reads like a budget for the whole conversation.
• 150 ms end-to-end at the mouth-to-ear level - the level below which the Recommendation says most applications will experience essentially transparent interactivity. Neither figure is a MUST-level requirement, and only the 400 ms carries the limit-shaped language.

A stop point can be placed where the agent starts speaking or where it finishes.

• First audio out - the first sample the caller can actually hear. The caller reacts to hearing something, so this is what governs perceived responsiveness.
• Complete response - the last sample of the agent’s turn. This governs when the turn ends and when the caller can act on the answer.
• A filler phrase splits the two - an agent that says “one moment” while it looks up an account has a fast time to first audio and an unchanged time to a usable answer.
• A suite recording only first audio reports an improvement the caller never received, and keeps reporting it as the work behind the filler gets slower.
• A suite recording only completion misses a genuine gain in how responsive the agent feels, and scores a filler phrase at zero.
• Report both against the same start point, so the distance between them is its own number.
• Name the endpoints inside the metric name - speech-end to first-audio and speech-end to answer-complete carry their reference points into any dashboard or status update.

The standards show the same instinct on a different parameter. G.107 (06/2015) permits only its predefined delay sensitivity settings, requires the Default class for carrier-grade and enterprise-grade telephony and whenever the target user group and delay requirements are unknown, and asks that any non-default use be stated explicitly when results are reported.

Published Delay Guidance

The closest published baseline for conversational speech timing is ITU-T G.114, the in-force 05/2003 edition. It was written for telephony transmission planning, so every figure quoted here is delay accumulated between two people on a call. Each carries its own measurement scope, and clause 1 records that this parallel treatment confused how G.114 was applied.

• 400 ms one-way - clause 4 recommends not exceeding this for general network planning, and glosses that planning scope as UNI to UNI, as illustrated in ITU-T Rec. Y.1541.
• 150 ms end-to-end - clause 4 also applies this to end-to-end delay, “mouth-to-ear” in the case of speech, and says most applications will experience essentially transparent interactivity below it.
• Below 100 ms - clause 1 notes that highly interactive tasks may be affected by delays below 100 ms, citing tests in Annex B of previous versions, which the 2003 edition does not carry. 100 ms is no G.114 objective.

Appendix II is informative guidance on one-way delay for voice over IP, added by Amendment 1, with no normative force. Its figures are stated for VoIP mouth-to-ear delay, the closest published match to a call routed through an agent.

• Intra-regional routes - for many intra-regional routes at 5000 km or less, VoIP users are likely to experience mouth-to-ear delays below 150 ms, an illustrative calculation from Appendix III/Y.1541 that assumes a well-engineered access network and no more than 12 routers.
• Expectations differ by call type - clause II.2 says mid-200 ms delays may not be a serious problem for long inter-regional calls, whilst it is critical that planners keep local and regional calls clear of such delays, because those are expected to be completely delay-transparent.

G.107, the 06/2015 in-force edition of the narrowband E-model, sets no single delay limit. Its Table 1 recommends fixed settings of delay sensitivity, sT, and minimum perceivable delay, mT, picked on conversational interactivity together with users’ sensitivity to the delay effect, and on whether the call is a business or everyday one.

• Default - sT = 1, mT = 100 ms. Note 7 to Table 3 permits only these predefined settings, and Default covers carrier-grade and enterprise-grade telephony and any unknown user group or delay requirement.
• Low - sT = 0.55, mT = 120 ms.
• Very low - sT = 0.4, mT = 150 ms.
• Where the classes agree - the delay impairment stays at zero while absolute delay sits at or below a class’s mT, so the classes separate only above 150 ms.
• Reporting your class - any non-default use should be stated explicitly with results.

An agent’s thinking time sits outside what these two Recommendations describe, and neither assigns a budget to it.

• Where the standard stops - the G.114 figures measure transmission delay between two humans on a call, and the G.107 settings model how a listener tolerates it.
• What I am adding - the step from mouth-to-ear transmission delay to an agent response target is my own extension, and G.114 is the one delay figure set a standards body has published and scoped for conversational speech.
• What your clock covers - your measurement window includes endpointing delay and the rest of what the caller sits through, which G.114 never measured. Record that scope beside the number it qualifies.

Percentiles Over Averages

A caller hears one turn at a time, and each turn either comes back fast enough to hold the conversation together or it stalls; the stalled ones are the ones they are left describing afterwards. A mean that sits inside target can coexist with a meaningful share of turns sitting far outside it.

Each turn produces one measurement under the same window, endpointing delay included, and all of them belong in the report.

• The spread - median, upper percentiles and the maximum together, so a reviewer can see how far the worst turns sit from the typical one.
• A count past your threshold - how many turns exceeded your threshold, stated as a count, with the total turn count beside it.
• The slow turns themselves - transcript, scenario, route and timestamps for the worst handful, because a tail that thin is often traceable to a single cause.
• Sample size beside every percentile - an unqualified percentile hides whether it came from a long run or a short one.

One slow turn in an otherwise quick call is absorbed by the caller; several slow turns in sequence read as an agent that has stopped working, even when both runs produce the same average. ITU-T G.114 (05/2003) makes a point of the same kind about one-way transmission delay: as guidance rather than a requirement, it notes in clause 4 that highly interactive tasks can be affected by delays much lower than the one-way values it discusses for general network planning. Reading that across to per-turn latency distributions is my own inference.

• Tail weight - how heavy the slow end is, since a long thin tail and a fat one behave differently under load.
• Clustering - whether slow turns arrive adjacent to each other inside a single call or scatter across many calls.
• Position - where in the call the slow turns land, since an opening turn and a turn mid-task are judged differently.
• Two humps - a distribution with two peaks often points to two code paths, so split the runs by code path and report a percentile for each.

A high percentile is estimated from only the slowest few turns in your sample, so it can move each time you add a turn.

The percentile you gate a release on is usually the highest one you track, which means the number doing the most work in that decision is the one supported by the fewest turns in the run. Nothing in the figure itself flags that, so it is worth knowing whether your harness counts the samples behind a verdict and withholds it while there are too few of them. What counts as enough is still yours to establish from your own runs, since it depends on what the decision costs you when the figure turns out to have been noise.

• The higher the percentile, the larger the run - you need enough turns that the extreme end is populated by many observations rather than by two or three.
• Name the turns behind the number - state how many of the slowest turns your top percentile actually rests on, so a reader can judge whether the figure sits on a handful of observations or on a solid block of them.
• Watch it settle - recompute the percentile as turns accumulate and see where it stops swinging; that point is your working minimum for a run.
• Slicing shrinks the sample - splitting by scenario, region or carrier divides your turns, and each slice needs its own count before its percentile is quotable.
• Repeat before you act - a percentile that shifts materially between two identical runs is a property of that sample, and the run needs to be longer before you gate on it.

Quote a percentile only with its sample size, its measurement window and the distribution shape attached.

Latency Under Concurrency

Measuring latency one call at a time, against a system with nothing else to do, reports close to the best your agent will do. Production latency is shaped by every session running beside it, so an isolated figure cannot predict what a caller gets at peak.

Concurrency adds time in places a solo call never touches:

• Contention for shared capacity - generation slots and speech workers a solo call had to itself are now competed for by every concurrent caller, and that wait sits inside the window you time.
• Timeouts and retries - a component that gives up and retries under pressure turns one ordinary turn into a very long one, and those land at the top of the distribution.
• Tail growth outrunning the median - the slow end moves first and moves further, so the average stays respectable while the worst turns get much worse.

Track p50, p95 and p99 at every step of the ladder. A median that holds flat as concurrency rises tells you little if p95 climbs over the same range, and the callers most likely to hang up sit in that p95.

Generating concurrent inbound call load is its own build, separate from the instrumentation that times a single call, and in practice it is what decides how high your ladder reaches. TestMu AI keeps performance and load testing as a separate surface behind a per-organization feature flag: it simulates concurrent inbound call workloads and reports per-band response time and success rate alongside breaking-point analysis, the concurrency level at which quality or availability degrades. Whatever places the calls, read any degradation level a harness reports against your own p95 at the same step, since the tail can start climbing well below the level where anything looks broken.

• Hold everything else constant - same script, same input audio, same persona, same region, same model version, same configuration down to the endpointing delay the window includes, so concurrency is the only variable that moves.
• Run a ladder - one call, then a level near the concurrency you expect at peak, then one above it, with enough calls per step for the percentiles to mean something.
• Report the curve - publish latency against concurrency for every percentile you track, so a reader can see where each one starts to climb.
• Mark the bend - record the concurrency where the curve stops being flat, because that is your practical limit, and it can arrive while error rates still look clean.

ITU-T G.114 (05/2003) supplies reference points for the curve, though neither is a pass mark for a voice agent:

• 400 ms one-way - G.114 recommends not exceeding this for general network planning, a scope it glosses as UNI to UNI, and phrases that as a recommended upper bound rather than a hard requirement.
• 150 ms end-to-end - mouth-to-ear in the case of speech, the level below which the Recommendation says most applications will experience essentially transparent interactivity, with its limit-shaped language staying on the 400 ms figure.

Those sit at different measurement reference points, so keep them apart when reading either against your window. G.114’s own Introduction records that stating network delay alongside application level mouth-to-ear delay led to confusion in how the Recommendation should be applied. It says nothing about voice agent stacks, so carrying its figures onto an agent turn is my own inference, sound only where your window covers the same mouth-to-ear path.

A load test reaches only part of what shapes production latency:

• Your stack only - a load test drives the components you run and the capacity you pay for, so the bend belongs to your deployment.
• Provider-side slowdown stays out of reach - if a speech or model vendor degrades under demand from other customers, your rig cannot create that condition on request.
• Production telemetry closes the gap - report the same percentiles from live traffic, and treat a shift the concurrency curve does not explain as something upstream.

Fields a Latency Report Carries

Someone will hold your latency number against theirs, or against the same suite after a later run, and act on the gap. Without the start and stop rules the comparison cannot be lined up, and without the run conditions nobody can reproduce the number.

ITU-T G.114 (05/2003) carries two delay figures at two different reference points, and its Introduction records that in earlier versions this parallel treatment of network and application level delays “led to confusion in how ITU-T Rec. G.114 should be applied”.

• 400 ms, one way - the value G.114 recommends not exceeding for general network planning, a scope it glosses as UNI to UNI, user-network interface to user-network interface.
• 150 ms, mouth to ear - the end-to-end level below which G.114 says most applications experience essentially transparent interactivity.
• Why the pair matters - quote either figure without its reference point and the reader compares it against the other.

An unrecorded change to the start or stop rule makes every earlier run incomparable, and on a chart that break looks like a regression. Declaring settings has precedent: ITU-T G.107 (06/2015) permits only its predefined delay sensitivity settings, reserves the default class for carrier-grade and enterprise-grade telephony and for any case where the target user group and delay requirements are unknown, and says any non-default use should be stated explicitly when reporting results.

• Definition ID - stamp it into the run artifact, not a wiki page, so number and rules travel together.
• Change record - note what moved and which runs fall on each side.
• Overlap run - re-run a known build under both definitions, giving the series a comparison point.
• Scope note - name the path and terminal the definition covers, since a rule for one leg does not carry to another.

When a number moves, check the measurement before you open the system, since a changed definition and a changed condition are the cheapest explanations available.

• Definition first - confirm both runs used the same start and stop rules and the same suite version.
• Conditions second - read the concurrency level and call path recorded on each run, not the ones the rig is set to now.
• Sample third - how many calls, at what time of day, and whether a failed batch thinned the run.
• System last - with definition, conditions and sample matching, the shift earns an investigation.

Conclusion

Write the start and stop points into the metric name before changing anything in the system. Most teams discover at that point that two dashboards have been measuring different things under one label.

Then report a distribution at a stated concurrency, since a single figure from an idle system is the best case rather than a prediction.

When a number moves, check the definition and the conditions before investigating the stack. Those are the cheapest explanations and, in my experience, the most common ones.