Background Noise Testing for Voice Agents

The Limits of a Noisy Clip

You already play a cafe recording behind your test audio, and the run feels like a noise test. How a full call is scored end to end is covered in testing AI calling agents. It returns a pass or a fail with no record of the condition behind it. Two weeks on, nothing on record says whether the noise sat where it sat before.

• The speech level - how loud the caller utterance is by the time it reaches the agent’s input.
• The noise level - how loud the cafe bed sits under that utterance in the same mix.
• The ratio between the two - signal-to-noise ratio, stated in decibels, the single number that sets how hard the condition is.
• The playback path - the gain staging, normalisation and re-encode steps between your audio files and the agent, each free to move between runs.

Two runs of the same scenario, with the same script and the same cafe file, become two different conditions the moment those levels move.

• A failed run - a real regression looks identical to a noise bed that landed louder than last week.
• A passing run - a genuine improvement looks identical to a quieter noise bed under an unchanged agent.
• A release-over-release trend - movement across releases carries both your change and the drift in your audio assets.
• A bug report - “fails in cafe noise” does not tell the engineer picking it up at what ratio it failed.

The controlled variable in a noise test is signal-to-noise ratio, the speech level minus the noise level in decibels, measured at the agent’s input.

• The clip is an asset - a cafe bed, a road bed, a keyboard bed, each carrying its own spectral and temporal character.
• The ratio is the condition - the same cafe bed at two different ratios gives two different tests, and both belong in the suite.
• The result travels with the ratio - a pass logged without its ratio cannot be compared to anything, including the same suite a month later.

Speech systems that carry noise suppression already have a published test methodology, and the shape of it is worth borrowing from. ITU-T Recommendation P.835 is titled “Subjective test methodology for evaluating speech communication systems and terminals providing noise suppression functionality”.

• What the title scopes - systems and terminals providing noise suppression functionality, a neighbouring problem rather than a specification for your agent suite.
• What the title says about method - subjective, meaning human listener judgement, so the procedure has no route into an automated CI run.
• Current edition - P.835 (07/26), approved on 29 July 2026 and listed by ITU-T as in force (prepublished).
• What it replaced - the 11/2003 edition, titled “Subjective test methodology for evaluating speech communication systems that include noise suppression algorithm”, now listed as superseded along with its amendment, erratum and corrigendum.
• What you borrow from it - the working posture that a noise condition is named and fixed before a run, which stays yours to build rather than anything readable out of a title.

Hold voice and accent constant while you bring the ratio under control, since speaker variation moves the same result. A run logged with its signal-to-noise ratio can be set against a target and against any earlier run at that same ratio.

Controlling the Ratio

A spoken clip carries pauses between phrases, breath gaps and silence at the head and tail of the file, so the speech level you measure depends on which samples you count as speech.

Average the power across the whole file and every silent stretch drags the figure down, which makes the stated ratio partly a function of how much silence the clip contains: two takes of one sentence, trimmed differently, mix to two different conditions under one label.

ITU-T P.56, “Objective measurement of active speech level”, standardises how the speech level itself is measured. The broader audio-quality picture is covered in voice quality testing.

• Method B - integrates a quantity proportional to instantaneous power over the aggregate of time during which the speech in question is present, the active time, and expresses total energy divided by active time in decibels.
• What it leaves out - noise that is not part of the speech, such as impulses, echoes and steady noise during periods of silence, rather than averaging across pauses.
• What it asks you to report - the activity factor alongside the level, and the margin M where a raised threshold is used in high-noise conditions.
• What it does not cover - the text defines an active speech level and an activity factor and carries no mention of signal-to-noise ratio, so the ratio and the reference point you build on that level remain yours to state and record.
• Edition - the wording above is from the 12/2011 edition, now superseded. P.56 (07/2026), approved on 29 July 2026, is listed in force (prepublished) and keeps the same title.

Two teams can both mix at what they call the same ratio and land on different conditions, because one measured across the whole file and the other measured active speech only. Your recorded conditions therefore carry the basis as well as the number.

• The speech level basis - active speech level per P.56, or a whole-file average, named explicitly.
• The activity factor - so a reviewer sees how much of the clip was speech.
• The margin M - where a raised threshold was used for a high-noise condition.
• The noise source and mix point - which generator, which material, and where in the chain the two signals were combined.

ITU-T G.191 specifies the ITU-T Software Tool Library. The ITU’s own openitu/STL repository copy of the G.191 main text, the STL2024 edition, a partial rendition that points to itu.int for the complete official text, summarises the library as source code for speech and audio processing modules for narrowband, wideband and super-wideband telephony applications, and states that the set includes codecs, filters and noise generators.

• P.810 MNRU - ITU-T P.810, “Modulated noise reference unit (MNRU)”, appears in that same copy among the normative references whose provisions constitute provisions of the Recommendation, and the STL keyword list names MNRU and P.50 FB MNRU among the tools.
• Scope of that citation - everything quoted here comes from the STL2024 text for G.191 (05/24), which ITU now lists as superseded by G.191 (07/26), and that text does not reflect the entire Recommendation.
• Why it helps you - with a standardised generator behind the generation step, a gap between two runs narrows to your mixing procedure, since both sides ran the same published code.

Noise That Actually Breaks Recognition

Noise interferes with recognition most when it occupies the same acoustic space as the voice underneath it. Competing speech shares the talker’s frequency range, syllable-rate modulation and pauses, so it covers what a recogniser leans on. Steady mechanical sound at the same level leaves more of it intact.

Separating a voice from a fan and separating it from other voices load the front end differently. The fan is stationary, so its spectrum can be estimated during quiet stretches and held. Other voices move the way speech moves, so the estimate keeps chasing them, and the system must also pick a talker.

• Steady noise - HVAC, road drone, a fan. The spectrum is stable, so estimation has something to lock onto, and accuracy tends to hold until the level climbs.
• Intermittent noise - a keyboard, dishes, a dog. What the front end learned during the quiet stretch goes stale when a burst lands, and how fast it re-estimates decides whether the straddling word survives.
• Speech-like noise - a television, open-plan babble. It shares spectrum and modulation with the target, so suppression tuned for steady noise starts taking the target with it.

• Passing one family says little about the others - a set built only from steady noise over-reports how well your agent copes.
• Fix the talker’s level the same way every time - ITU-T P.56, “Objective measurement of active speech level”, specifies a Method B that integrates a quantity proportional to instantaneous power over the active time, the aggregate of time during which speech is present.
• Relating that level to the noise is your own convention - P.56 never mentions signal-to-noise ratio, so record how you set the noise side and hold it constant.
• Hold the confounds still - vary the noise family while keeping the same speaker, accent and script, or voice differences absorb the result.

ITU-T G.160 “Voice enhancement devices” (edition 06/12, approved 2012-06-29, in force) is the Recommendation that carries the objective characterization of noise reduction algorithms.

• What Appendix II gives you - an objective methodology for characterizing the basic effect of noise reduction methods, with reference ANSI-C source code, plus objective measures for characterizing NR solutions.
• Appendix II is informative - adopt it as a methodology; it gives you no normative requirement to hold a vendor to.
• G.160’s scope is wider than noise reduction - it also addresses acoustic echo control and TFO/IWF signal handling, and explicitly does not contain objective tests of the voice quality performance of noise reduction functions.

P.56’s Method B excludes impulses by design, so the level measurement says nothing about what a brief loud event costs. A door, a horn or a cough is over inside a second, and its onset resembles the start of speech closely enough to be read as a turn boundary.

• Place transients deliberately - one before the user speaks, one mid-phrase and one on the final word, since a clipped first word and a missing tail are different failures.
• Report direction before magnitude - expect the ordering across noise families to travel between setups better than the size of the gap does, since how much a family costs depends on your front end, model and codec path.

Noise Suppression Cuts Both Ways

Suppression removes background sound by deciding which parts of the signal are not speech, and that classification is wrong some of the time. When it errs in one direction, speech energy leaves with the noise.

• What goes missing first - low-energy consonants, word onsets and trailing syllables sit closest to the noise floor, so an aggressive setting reaches them first.
• Why it hides - the audio still sounds clean to a reviewer while the transcript loses word endings, so the investigation starts at the model and the signal chain goes unchecked.
• The quiet-room case - a caller in a silent room gives suppression almost nothing to subtract, and an aggressive profile still acts on the speech.

A caller in a quiet room can be recognised worse with suppression on than with it off. Teams find this late because their noise suite only exercises noisy inputs, where suppression tends to improve the result it touches.

Characterising the basic effect of a noise reduction method is already standardised work.

• The document - ITU-T G.160, “Voice enhancement devices”, edition 06/12, approved on 29 June 2012 and in force.
• What Appendix II gives you - an objective methodology for characterising the basic effect of noise reduction methods, with reference ANSI-C source code, and three objective measures for characterising noise reduction solutions.
• The status caveat - Appendix II is informative, which makes its methodology optional for a vendor and unusable as a conformance claim.
• The scope caveat - G.160 also covers acoustic echo control and TFO/IWF signal handling, and explicitly does not contain objective tests of the voice quality performance of noise reduction functions.

Suppression settings are an input to every noise result you record. A pass rate recorded without them cannot be reproduced, because the next run may apply a different profile to the same clips.

• Vendor and version - the suppressor identity and build, since behaviour can change between releases and a build number is the only way to tell.
• Mode and aggressiveness - the named profile and any strength parameter, captured as configuration in the run record, where a diff shows when it changed.
• Position in the chain - handset, carrier network, media server or model front end, and whether two stages run at once.
• Source level - the active speech level, measured per ITU-T P.56, whose Method B integrates a quantity proportional to instantaneous power over the active time and excludes noise that is not part of the speech.
• What P.56 asks you to report - the activity factor alongside the level, and the margin M where a raised threshold is used in high noise.

Quiet audio with suppression enabled is a cell you have to keep, and the one easiest to drop, because it feels like testing nothing.

It is also the cell most likely to pass on task outcome while sounding worse, so it rewards a gate that can be set on functional criteria, on audio quality, or on both. Separating those two verdicts is what stops a suppression regression in a quiet room from hiding behind a completed task.

• Quiet, suppression off - the reference condition other cells are measured against.
• Quiet, suppression on - the same utterances, same speaker, same accent, same level, with the production profile applied.
• The assertion you write - recognition of a clean signal does not degrade when suppression is on.

A negative test here catches over-aggressive configuration before a release reaches callers who happen to be somewhere quiet. A failure points at the profile before the model, which is the cheaper place to start.

Reading the Result

Read a noise sweep as a curve across levels. A single run at a single level gives you a point and no shape. The output worth acting on is the level at which the agent stops completing the task.

A sweep produces a second curve worth reading beside it, which is how the call sounded rather than whether it worked. P.835 itself stays a listening-panel method, so what a pipeline can carry is a score computed from the audio instead of from listeners. TestMu AI’s agent testing platform scores every inbound call for DNSMOS P.835 from the post-call recording, with no transcript needed and nothing changed about how the call ran, and separates Overall from Signal and Background.

• One moving part - hold the scenario, the speaker and the accent constant across the sweep so the only thing changing between runs is the noise condition.
• Find the transition band - the level where completion starts to slip and the level where it stops entirely need not be the same, and the distance between them tells you how gradual the decline is.
• Note the width - a wide band suggests the agent has room to recover and reprompt, while a narrow one points to the agent losing the task over a small change in conditions.
• Repeat near the edge - runs close to the transition can vary between attempts, so several runs per level give you a completion rate you can compare instead of one anecdote.
• Compare curves between builds - the regression signal you want is the transition moving toward easier conditions, which a per-run pass or fail at a single level will not surface reliably.

Record whether the agent completed the task, then use the transcript to explain the runs that did not. An agent can mis-transcribe several words in an utterance and still route the call correctly, because the intent survives the damage. The reverse happens too, so score the end state before you score the words.

• Score the end state first - whether the agent reached the outcome the scenario called for, with the right values captured, before you look at anything else.
• Keep partial successes separate - a task that completed only after reprompting is a different result from one that completed on the first pass, and merging them hides the degradation you are hunting.
• Treat measured accuracy as a covariate - use it to explain the shape of the completion curve, and keep completion rate as the result you compare between builds.

When a noisy run fails, the outcome alone will not tell you where the failure started. Pull the transcript for every failed run and match it against the end state before you file the cause.

• Recognition - the words arrived wrong, and the transcript shows the substitution or deletion that carried the turn into the wrong intent.
• Endpointing under noise - the transcript is clipped early or fused with the following turn, which points at the turn boundary, so check endpointing before you suspect recognition.
• Low-confidence handling - the transcript is close enough to act on, and the agent still reprompted, escalated or stalled, which puts the fault in the agent’s own policy for uncertain input.

Keep the measured condition attached to every result you record. ITU-T P.56, “Objective measurement of active speech level”, asks that the activity factor, and the margin M where a raised threshold is used in high-noise conditions, be reported alongside the level itself. That request is scoped to the active speech level measurement, so the equivalent habit for your noise conditions is one you impose on your own logs.

Building the Noise Set

Treat the noise set as a fixture you version alongside the suite. Every condition below stresses a different stage, and the set stays small enough to run on every build.

Record every condition at one sample rate and bit depth, and mix each one against the talker at a fixed active speech level. Both travel with the fixture, so a clip added next quarter matches the originals.

• Record real rooms - capture the coffee shop, the car cabin and the call center bay, because a recorded room brings the room, the talker distance and the capture chain with it. Keep the STL’s generated conditions in the set as the calibration anchor, and let the room recordings carry the coverage.
• Set the mix against active speech - ITU-T P.56, “Objective measurement of active speech level”, defines the measurement. Its Method B integrates a quantity proportional to instantaneous power over the aggregate of time during which the speech is present, the active time, and expresses total energy divided by active time in decibels, deliberately excluding noise that is not part of the speech, such as impulses, echoes and steady noise during periods of silence. Levels stop drifting with how much the talker pauses.
• Write the mix parameters into the result - P.56 asks for the activity factor alongside the level, and for the margin M where a raised threshold is used in high-noise conditions. P.56 never mentions signal-to-noise ratio, so any ratio you quote is your own convention on top of the level measurement it defines.
• Hold the talker constant - same speaker, same script, same accent, same microphone and same capture chain across every condition, so the noise is the only variable that moved.

For an outside anchor, ITU-T P.835 covers subjective test methodology for evaluating speech communication systems and terminals providing noise suppression functionality. Its 07/2026 edition supersedes the 11/2003 edition and is in force (prepublished).

ITU-T G.160, “Voice enhancement devices”, covers more than noise reduction: it also addresses acoustic echo control and TFO/IWF signal handling, and it explicitly does not contain objective tests of the voice quality performance of noise reduction functions. Its Appendix II presents an objective methodology for characterizing the basic effect of noise reduction methods, with reference ANSI-C source code and three objective measures for characterizing NR solutions. Appendix II is informative, so nothing in it is a conformance requirement to pass.

Pruning runs off recorded pass rates, so log a verdict per clip per build.

• Checksum every clip - store the set in the repository next to the suite, so a re-encode shows up as a diff instead of as a score change.
• Keep the conditions that discriminate - keep a condition when its verdict has flipped at least once across the last quarter of builds, since a flip is evidence it can still catch a regression.
• Archive a condition that has stopped moving - when every build passes a condition, or every build fails it, that condition has stopped carrying information and should leave the gate.
• Re-check on a fixed cadence - review pass rates per condition each quarter as your own review rhythm, and bring an archived clip back when the front end or model changes enough to matter.

Conclusion

Start by recording the ratio you are already testing at, even before you change anything else. Most teams find they cannot answer the question, and that answer alone explains a run of results nobody could reconcile.

Then fix the measurement method and write it down beside the number, because a ratio computed two different ways is two different conditions wearing one label.

Add the quiet control last. A clean signal that degrades when suppression is enabled is the defect nobody goes looking for, and it takes one scenario to catch.