Accent Testing for Voice Agents: Who Gets Understood

The Shape a Pooled Average Hides

Koenecke et al., writing in PNAS in 2020, tested five commercial speech recognition services and reported an average word error rate of 0.35 for Black speakers against 0.19 for white speakers. Those figures describe five vendor APIs as they stood in 2019, which is the claim the measurement supports.

• The systems tested - Amazon, Apple, Google, IBM and Microsoft, through the vendors’ 2019-era APIs.
• The sample - 2,141 Black and 2,141 white audio snippets, 19.8 hours of audio from sociolinguistic interviews at five US sites, nearest-neighbor matched on speaker age, gender and snippet duration.
• Best of the set - Microsoft’s system had the best overall performance, at 0.27 (SE 0.004) for Black speakers against 0.15 (SE 0.003) for white speakers.
• Worst of the set - Apple’s had the worst overall performance, at 0.45 (SE 0.005) against 0.23 (SE 0.003). The authors note its streaming, real-time transcription, not used by the other four, may account for part of that result.
• The corpora - Black-speaker audio from CORAAL, white-speaker audio from Voices of California.

How a full call is scored end to end is covered in testing AI calling agents. An aggregate accuracy number is a weighted average over whoever was in the sample, so each cohort influences it in proportion to its share of the audio. Set your headline figure beside a per-cohort breakout before you read it as a quality signal.

• Share sets the influence - a cohort’s share of the audio is its weight in the average, so its own error rate is discounted by that share before it reaches the top line, however severe its failures are.
• A failing cohort barely moves it - a small group failing badly leaves the headline number close to where it started.
• One number flattens the shape - a pooled average settles between the cohort rates and reports neither, the way 0.35 and 0.19 collapse into a single figure.
• What a breakout showed - in the same study, error averaged across the five systems ran 0.41 for Black men, 0.30 for Black women, 0.21 for white men and 0.17 for white women.
• How to report it - word error rate per cohort, with the sample size beside it.

The matched subset is the part to copy into your own test design.

• What they built - 206 matched utterances of 5 to 8 words with identical ground-truth human transcripts, spoken by Black and white interviewees of the same gender and approximately the same age.
• What it showed - across all five systems, word error rates ran roughly twice as high for the Black speakers, with Microsoft at 0.13 against 0.07.
• Why the design matters - both sides say the same words, such as “and then a lot of the”, so the remaining gap cannot be explained by people saying different things.

On where the gap comes from, the authors point primarily at a performance gap in the acoustic models, with their language-model check coming back clean.

• The language-model check - perplexity under Transformer-XL, GPT and GPT-2, used as proxies for commercial language models, found the Black speakers’ snippets no harder, with lower perplexity in fact.
• What they attribute it to - pronunciation and prosody, including rhythm, pitch, syllable accenting, vowel duration and lenition, rather than the grammatical or lexical characteristics of African American Vernacular English.

Choosing the Cohorts

Acoustic conditions are a separate axis, covered in background noise testing. A cohort is any group of speakers you are willing to report a separate number for. Splitting on it earns its place when the split changes what your callers get, and the axes that survive are the ones you can recruit for and describe to a stakeholder without a glossary.

• Regional variety - speakers of the same language from different places, which is usually the easiest axis to source and the easiest to explain.
• First-language background - English spoken by people whose first language is Tagalog, Spanish, Hindi or Mandarin, grouped by that background rather than by nationality.
• Speaking rate - fast and slow talkers, an axis that applies inside any regional or first-language group and is worth splitting out before you file a failure under accent.

Labels are coarser than the thing they stand for, so expect variation inside a cohort. Koenecke et al. (PNAS, 2020), testing five commercial systems as they performed at the time, reported median per-snippet word error rate by interview site, each snippet’s rate first averaged across those five systems.

• Princeville NC 0.38, Washington DC 0.31, Rochester NY 0.20 - the three African American English interview sites in that study, a spread sitting inside a single cohort label.
• Sacramento CA 0.18, Humboldt CA 0.15 - the study’s two white-speaker sites, drawn from a separate corpus and a separate speaker population, with Rochester landing close to both.
• Dialect density, hand-coded for the Black speakers only - averaged 0.19 in Princeville, 0.088 in Washington DC and 0.047 in Rochester, an ordering that matches the word error rate ranking across those three sites.
• Where that gradient is established - within the three African American English sites rather than across all five locations, since Rochester came out comparable to the two California sites.

• Start from your own call volume - the cohorts worth naming are the ones with real callers behind them, which your logs and your support queue already tell you.
• A cohort with nobody in it - costs recruiting effort, adds a column to every report and predicts nothing about your traffic.
• The actual exposure - a large caller population you have never measured separately, sitting inside an overall accuracy figure that looks fine.
• Cross an axis when volume allows - Koenecke et al. found word error rates, averaged across the five systems, highest for Black men at 0.41, against 0.30 for Black women, 0.21 for white men and 0.17 for white women, so a second split can surface a subgroup the cohort average hides once each cell holds enough speakers to report.

You need enough distinct speakers per cohort that one unusual voice cannot define the group.

• One recording per accent - produces a figure that moves entirely with that person and their mood on the day.
• Prefer breadth over depth - spend a fixed recording budget on more speakers with fewer prompts each, rather than one speaker covering every prompt.
• Report the speaker count - next to every cohort number, so nobody reads a small-sample result as a population estimate.
• Hold recording conditions constant - across cohorts, so a gap you find is about the speaker rather than the capture setup.

Measurements Worth Taking per Cohort

Transcript accuracy tells you where the recognizer drops words and which cohort it struggles with. Task outcome is what the caller lives with, and it moves independently of the transcript:

• Transcript errors, correct routing - the agent mangles the middle of a balance request and still routes it correctly, because the intent classifier had signal left.
• Clean transcript, wrong outcome - the words come through intact, and the agent applies the payment to the wrong account anyway.

Log these fields per cohort on every run, alongside the run-level totals:

• Task completion - whether the caller got what the scenario says they came for, judged against the intent you assigned and the backend record, whatever the agent reports.
• Reprompt count - how often the agent asked the caller to repeat, rephrase or spell something, per turn and per call.
• Escalation rate - how often the call went to a human, split by whether the handoff followed recognition failures or a policy limit.
• Abandonment - how often the caller hung up mid-task, with the turn index where they left.
• Run metadata - model version, prompt revision, telephony path and codec, recorded on the row so a later reader knows which stack produced it.

A cohort can sit inside an acceptable accuracy figure while a meaningful share of its callers are asked again for an account number, because each retry lands and the task completes. Reprompt count per cohort surfaces that pattern, and completion rate absorbs it.

Koenecke et al. (PNAS, 2020) reported median per-snippet word error rate, each snippet averaged across five commercial systems from Amazon, Apple, Google, IBM and Microsoft, by interview site.

• Three CORAAL African American English sites, Black speakers - 0.38 in Princeville NC, 0.31 in Washington DC, 0.20 in Rochester NY.
• Two Voices of California sites, white speakers - 0.18 in Sacramento, 0.15 in Humboldt.
• The authors’ own scoping - Rochester’s 0.20 is comparable to the two California figures, so the gradient in dialect density, how heavily dialect features show up in speech, holds within the three CORAAL sites alone.
• Where to set the boundary - one broad label spans 0.38 in Princeville down to 0.20 in Rochester, so define cohorts at a grain you can act on.

Those figures describe those five services as they tested in that study. Vendor accuracy on your traffic today is what your own measurements answer.

Comparing inside one run matters more than the absolute figures, since a model or a pipeline change between runs moves every cohort at once. A suite that expands one scenario across a set of voices in a single pass gives you that for free; building it by hand means pinning the script, the synthesis and the capture path yourself and re-pinning them every time one moves.

Compare cohort against cohort inside one run, and set reporting up like this:

• Hold the run constant - same model version, prompt, telephony path and recording conditions, then read the spread between cohorts.
• Report the gap alongside the level - the ratio between your best and worst cohort is the more durable of the two across a model upgrade, so trend it.
• Re-baseline every run - the absolute level moves with the model version, codec and prompt, so an earlier run’s cohort score is not comparable to this one.
• Watch the rank order - if the worst cohort changes identity between runs, something in the stack shifted, and only the per-cohort split surfaces it.

Put completion, reprompts, escalations and abandonment side by side per cohort in every run report. Order it by gap size, so the worst cohort and its distance from the best read off the page.

Separating Accent From the Channel

A cohort gap is only evidence about accent when both sides of the comparison traveled the same audio path. If one group was captured on a headset and another on a speakerphone over a congested mobile leg, part of what you measure belongs to the equipment. Anything differing between cohorts other than the speaker is a competing explanation, so hold these fixed.

• Capture path - the same microphone class, device and sample rate for every speaker, captured before any cleanup stage runs.
• Codec and transport - the same encoding, bitrate and network conditions, since compression discards different detail at different settings.
• Acoustic conditions - the same room and the same background level for every cohort, held fixed as a control.
• Scenario content - the same prompts, utterance lengths and vocabulary, so that one cohort is not handling harder words than another.

A stitched corpus can carry more than one channel, since each source was collected on its own terms. Comparing cohorts across it measures the sources as much as the speakers.

Generating both sides of the comparison through one path avoids that, which is why it is worth checking whether your harness records the conditions each call ran under. TestMu AI’s agent testing platform persists the resolved voice, noise preset and persona on every call, so a cohort difference can be traced to the speaker rather than to a setting nobody wrote down.

• The sources differed by region - Koenecke et al. (PNAS, 2020) drew Black-speaker audio from CORAAL sites in Princeville NC, Washington DC and Rochester NY, and white-speaker audio from Voices of California in Sacramento and Humboldt. The authors note the two groups came from different regions of the country.
• Matching covered the speakers - 2,141 snippets per group, about 19.8 hours of interviewee audio, matched by nearest neighbor on speaker age, gender and snippet duration, which covers speaker attributes and clip length while leaving the recording channel unmatched.
• A controlled-text subset - 206 utterances of 5 to 8 words with identical ground-truth transcripts, spoken by people of the same gender and approximately the same age, still showed word error rates roughly twice as high for the Black speakers across all five systems, 0.13 against 0.07 on Microsoft, the best overall performer. Those numbers describe Amazon, Apple, Google, IBM and Microsoft as they performed in that 2020 study, and any of those systems may behave differently today.

Holding the words constant removed content as a variable, which led the authors to conclude the disparity arises primarily from pronunciation and prosody rather than from grammar or vocabulary. That comparison ran between Black and white speakers in US English sociolinguistic interviews, which scopes the finding to dialect cohorts inside one language variety.

• Fixed script - every cohort speaks the same utterances, including the same names, digits and product terms.
• Fixed rig - the same capture path and codec for all of them, logged per recording.
• Single variable - the speaker changes and nothing else in the setup does, checked against the recording log before analysis.
• Stated caveat - when a cohort can only be sourced elsewhere, report the channel difference alongside the gap.

A gap produced under those controls still needs interpretation, since a word error rate says nothing about which errors changed the outcome for a caller. The channel is at least accounted for, so the next question can be which sounds the model is missing.

Reading the Gap

Recognition, understanding and low-confidence handling each produce the same cohort spread in your outcome metric. Which one is at work decides whether you touch the recogniser, the NLU or the fallback policy.

• Recognition - the transcript is wrong, so every step after it reasons over words the caller never said.
• Understanding - the transcript is right and intent classification or slot filling still misses, often on phrasing common in one cohort and thin in your training data.
• Low-confidence handling - scores run lower for one cohort, and your fallback policy turns that into extra re-prompts, more confirmations or an earlier transfer.

Koenecke et al. (PNAS, 2020) ran an attribution exercise one level down, inside recognition itself, across the five commercial systems they tested. They compared 206 matched utterances of 5 to 8 words with identical ground-truth text.

• Language models cleared - perplexity under Transformer-XL, GPT and GPT-2, used as a proxy for commercial language models, came out lower for Black speakers’ snippets than for white speakers’, so the text itself was no harder.
• Error rates still doubled - word error rates on those identical phrases stayed roughly twice as high for Black speakers across all five systems, with Microsoft, the best overall performer there, at 0.13 for Black speakers against 0.07 for white speakers.
• Acoustic modelling named - the authors put the disparity primarily on the acoustic models, pointing to pronunciation and prosody: rhythm, pitch, syllable accenting, vowel duration and lenition.

That conclusion belongs to those five systems as they performed for that study in 2020. Your own stack needs its own attribution, and a smaller version of the same comparison gets you there: hold the transcript and the outcome side by side per cohort.

• Transcripts wrong, outcomes wrong - recognition is doing the damage, so recogniser choice, custom vocabulary and pronunciation hints are live options.
• Transcripts right, outcomes differ - the problem sits downstream of recognition, and swapping recognisers will leave the spread where it is.
• Same result, longer path - when one cohort gets there after more turns, inspect confidence thresholds and fallback rules before touching any model.

Whether a cohort disparity blocks a release is a policy call, and the measured number arrives with no threshold attached.

• Set the tolerance first - agree the maximum acceptable spread between best and worst cohort while nobody yet knows which cohort will be worst.
• Name who can accept a gap - someone must be able to ship a known disparity deliberately, with a remediation date attached.
• Keep the reasoning - store the argument for shipping beside the number that prompted it, because the next release reopens the same debate.

An upgrade moves cohorts in different directions, so an improved headline figure can sit on top of a cohort that went backwards.

• Same set, both versions - replay the identical cohort set before and after, then read the per-cohort numbers, because an aggregate hides one cohort rising while another falls.
• Judge by the worst cohort - gate the upgrade on the weakest cohort’s number, so a better average cannot carry a regression through.
• Hold conditions constant - keep recording conditions identical across cohorts and runs, so any movement in the number can only have come from the model.

Acting on a Measured Disparity

A cohort gap comes from recognition, intent matching, confidence handling, or the test set itself, and each takes a different repair. Score intent on corrected transcripts first, since that separates a recognition problem from an understanding problem.

Swapping the recognizer is the obvious first move. In the 2020 PNAS study by Koenecke and colleagues, on matched conversational-interview snippets run through 2019-era APIs, Microsoft performed best of the five systems tested at 0.27 word error rate for Black speakers against 0.15 for white speakers, and Apple performed worst at 0.45 against 0.23, which the authors note may owe partly to Apple’s streaming transcription. That swap moved absolute error rates a long way and left the cohort ratio near two-to-one.

• Recognizer configuration - cheapest to try; re-measure every cohort after each hint-list change, since a hint tuned for one cohort can raise error for another.
• Prompt and confirmation behavior - shorter turns and explicit spoken-or-keypad choices cut re-prompts per completed task; cohort word error rate is a separate number on its own schedule.
• Targeted audio in the evaluation set - more speakers per thin cell narrows the interval, so the next measurement is tight enough to act on.
• Earlier human routing - drop the escalation threshold for the cohort the system handles worst, and watch agent minutes per call as the cost side.
• Routing is temporary - log the lowered threshold with an expiry date and the cohort word error rate attached, since recognition error for that cohort is unchanged.

A bigger or newer model moves cohorts by different amounts, so an aggregate gain can sit on top of a regression for one group. In the Koenecke sample, 206 matched utterances of 5 to 8 words with identical text still produced error rates roughly twice as high for Black speakers across all five systems, pointing at the acoustic model rather than the language model.

Tie cohort measurement to the changes that shift the distribution.

• Model or vendor change - rerun the cohort set before rollout, since the aggregate can improve while one cohort regresses.
• Prompt or dialogue change - re-measure the cohorts whose confirmation and re-prompt rates were already highest.
• Quarterly with nothing shipped - your caller mix drifts anyway, and a test set built last year ages against it.

Conclusion

Break out the number you already have before collecting any new audio. Most teams can split their existing test set at least once, and that first split usually answers whether there is a problem worth funding.

Then fix the capture path before comparing cohorts, because a corpus assembled from different sources measures the sources as much as the speakers.

Decide what size of gap would change a release decision while the number is still unknown. That threshold is much harder to set honestly once you are looking at the result.