How to Test a Deepgram Voice Agent

What Deepgram Voice Agents Actually Are

Deepgram spent its first several years as a pure speech AI infrastructure vendor, selling Nova speech-to-text and, more recently, Aura text-to-speech as APIs that other companies wired into their own agent stacks. The Voice Agent API changes that positioning: Deepgram's own general-availability announcement says the company now controls all three core components itself, Nova-3 for speech-to-text, Aura-2 for text-to-speech, and a "built-in orchestration" layer that governs turn-taking, tool use, and knowledge retrieval, instead of building a separate agent product on top of someone else's models.^[1]

Technically, a Deepgram Voice Agent is one WebSocket connection configured with a single Settings message split into three sections, according to Deepgram's own configuration docs.^[2]

• listen: the Nova speech-to-text model, for example nova-3.
• think: the reasoning LLM, which can be an outside provider such as OpenAI or a model Deepgram hosts, plus the system prompt and temperature.
• speak: an Aura text-to-speech voice, for example aura-2-thalia-en.
• audio: input and output encoding, sample rate, and container format for the raw audio stream.

That single-API design is the whole pitch: instead of running Nova for transcription, a separate LLM for reasoning, and a separate TTS vendor for output, then reconciling the latency and state mismatches between them yourself, Deepgram runs all three inside one connection it controls end to end. On top of that core pipeline, agents can be configured with function calling to reach external tools mid-conversation, multiple agents that hand off based on intent, and saved configurations you reuse across projects.

This is meaningfully different from a menu-driven phone tree. If you are unfamiliar with that distinction, see IVA vs IVR; a Deepgram Voice Agent is closer to infrastructure a developer assembles and configures via API than a product a non-technical user builds with dropdowns. That is also who tends to build on it: teams that want an integrated, low-latency voice pipeline from a single infrastructure vendor with a multi-year transcription track record, rather than assembling STT, an LLM, and TTS from three different companies and owning the glue code themselves.

How Deepgram Agents Fail in Production

Deepgram's own Nova-3 announcement claims a median streaming word error rate of 6.84%, a 54.2% improvement over the next-best competitor's 14.92%, and a median batch word error rate of 5.26%, a 47.4% improvement over a 10% competitor benchmark.^[3] Those are strong transcription numbers. They also, by Deepgram's own admission when introducing its Voice Agent Quality Index, do not predict whether a caller stays on the line: callers rarely hang up because a voice bot mistranscribes a word, Deepgram says; they quit because the conversation just feels "annoying."^[4]

Function-calling and tool orchestration errors. Function calling runs through two message types: a FunctionCallRequest the server sends when the agent decides a tool is needed, and a FunctionCallResponse the client, or Deepgram itself for server-side functions, returns with the result. A client_side flag decides who executes the call, per Deepgram's function-calling documentation.^[5] That handoff is newer than Nova's multi-year transcription track record, and it is where a wrong parameter, a timeout, or a mismatched client_side assumption produces a call that transcribed perfectly but still did the wrong thing.

Turn-taking and latency at the orchestration layer. Stitching listen, think, and speak into one pipeline does not remove the latency chain, it just moves who owns it. Deepgram built Flux, its conversational speech recognition model, specifically because pipelines built on separate ASR, voice-activity-detection, and endpointing layers running sequentially caused agents to either cut callers off mid-sentence or leave robotic pauses; Flux claims to cut agent response latency by 200 to 600 milliseconds versus that older pipeline approach and reduce false interruptions by roughly 30%.^[6] If your agent is not on Flux, or your think or tool layer is slow, you inherit the exact problem Flux was built to fix.

Transcription edge cases that survive a strong benchmark. Nova-3's headline WER is measured on general audio, not your product's account numbers, brand names, or industry jargon. Deepgram's own answer to that gap is Keyterm Prompting, which lets you boost up to 100 domain-specific terms at inference time so the model recognizes them in context.^[7] If you have not configured it, accented speech, crosstalk, and unusual vocabulary are exactly where a benchmark-beating model still mishears the word that mattered most in the call.

What to Test: The Four Dimensions

None of the four dimensions below is word error rate. WER, and its TTS equivalent, measures whether Deepgram transcribed or synthesized audio correctly in isolation; it says nothing about whether the resulting conversation actually helped the caller, stayed safe, or held up under a bad connection. Score every Deepgram Voice Agent against these four before it goes live.

How to Test a Deepgram Voice Agent, Step by Step

Testing a Deepgram Voice Agent means exercising the same WebSocket connection and message types your production client uses, not guessing from the docs. The steps below apply whether you test directly against the Agent API or through a phone bridge.

• Connect and configure. Open a WebSocket connection to the Voice Agent API and send a Settings message with your listen, think, and speak configuration, the same one your production agent uses.
• Exercise function calls deliberately. Script utterances designed to trigger each tool the agent can call, then confirm the FunctionCallRequest arrives with correct parameters and, for client_side calls, that your FunctionCallResponse is accepted. Test both client-side and server-side execution paths separately, since they fail differently.
• Capture transcript and audio per turn. The same connection surfaces the agent's transcript and synthesized audio; pull both so failures get scored against what the caller actually heard, not just what the log says was said.
• Test the telephony bridge separately from the raw API. If the agent runs behind a phone number, the audio path adds a codec conversion, base64 mulaw for Twilio, and barge-in handling, where the server sends a "clear" event to stop agent audio the moment the caller starts speaking.^[8] An agent that passes clean WebSocket tests can still misbehave once that bridge is in the loop.
• Layer in accents, noise, and adversarial personas. Clean, scripted audio will not surface the failures described above; run the same scenarios with background noise, interruptions, and callers who do not follow the happy path.

Running that by hand for every prompt or model change does not scale. TestMu AI's Agent Testing platform connects to a Deepgram agent's endpoint the same way a real caller would, then deploys autonomous evaluators that auto-generate 60-100+ scenarios from an uploaded prompt or document and score every run against 9 conversation-quality metrics, hallucination, bias, completeness, context awareness, and tone among them, plus 30+ call-specific metrics such as first call resolution and intent recognition accuracy once the agent is deployed on a phone number. It also simulates 200+ voice profiles across 50+ accents and 15 background-noise environments, so the accent and crosstalk gaps from the section above get tested before a real caller finds them.

The platform's Metric Thresholds panel sets the minimum score each metric must reach to pass, bias detection, hallucination detection, completeness, and context awareness among them, the same conversation-quality dimensions a Deepgram agent's transcript and audio get scored against. A run's summary surfaces average latency, a voice-quality score, and words per minute, with the caller and agent turns captured as separate audio waveforms so a reviewer can hear exactly where a call went wrong. The Testing Your First AI Agent docs walk through connecting your own agent endpoint end to end, and note that for a voice bot, the platform "can test it by automatically generating audio for the test cases and evaluating the transcribed audio responses."

Why Manual and Single-Prompt Testing Isn't Enough

A single test call, even a well-scripted one, tells you the agent can handle one path through one conversation. Production callers do not follow scripts: they change their mind mid-sentence, interrupt, ask an off-topic question, then return to the original request three turns later. A Nova-3 benchmark score describes performance on a fixed, usually clean, audio set; it cannot describe what happens when the same acoustic model has to track state across eight turns while a tool call executes in the middle.

This is where the earlier point about the Voice Agent Quality Index matters most: interruption handling and missed response windows carry the largest share of that score, precisely because they only show up across a real back-and-forth, not a single utterance. Testing with adversarial personas, an impatient caller, a confused one, someone who interrupts, someone with a heavy accent, catches the multi-turn failures a single happy-path prompt test will always miss. It is the same reason AI agents for SDETs increasingly treat agent evaluation as a scenario-coverage problem rather than a single pass or fail check.

Persona and Accent Coverage at Scale

The step-by-step run above works for a handful of scripted calls. What a small team physically cannot do by hand is reproduce the range of callers a Deepgram agent meets once it is live: the accents, the emotional states, the background noise, and the many ways people phrase the same request. That breadth is what scale actually buys, and it is the difference between testing the agent you imagined and the callers it will really get.

• A synthetic persona library reproduces callers a small team cannot stage on demand, 200+ voice profiles spanning 50+ accents and dialects, alongside emotional states from confused to impatient to hostile, run against the same agent in a single pass.
• One uploaded prompt or support document auto-generates 60-100+ scenarios spread across happy paths, edge cases, and off-script callers, far more conversation paths than any engineer writes by hand for one release.
• 15 background-noise environments, a call center floor, a moving car, a poor cellular connection, put the listen layer under the acoustic conditions callers actually dial from, not the clean audio a benchmark WER is measured on.

For teams comparing which language model to put in the think section of their agent, our guide to LLM agent frameworks covers the orchestration trade-offs, and The Voice Agent's Core Dilemma if you are still deciding how much conversational flexibility your use case actually needs.

Load and Concurrency Testing for Deepgram Agents

Persona variety tells you if the agent handles different callers. It doesn't tell you what happens when 200 calls hit the same phone number in the same five minutes. The WebSocket connection, the think-layer LLM, and the telephony bridge all have to hold up under that load, not just under one clean test call.

Agent Testing runs 5 parallel calls per org by default. Need more? Additional parallel capacity is available as a top-up, so you can simulate an actual concurrent spike instead of a queue of calls running one after another. There's no cap on total call volume either, parallelism just controls how fast a batch finishes. If your agent sits behind a campaign or a support line that spikes, that's the number to plan test coverage around.

Red-Teaming Deepgram Agents for Jailbreaks and Prompt Injection

The safety dimension above covers a caller trying to get help. Red-teaming covers a caller trying to break the agent on purpose: talk the think-layer LLM into ignoring its system prompt, pull data through a function call it shouldn't return, or say something that gets compliance on the phone.

testmu-a2a-cli runs this as a dedicated command against the agent's endpoint, scoring nine attack categories, prompt injection, jailbreak attempts, data exfiltration, PII leakage, harmful content, overreliance, hijacking, policy violation, technical injection, at basic, intermediate, or advanced intensity:

testmu-a2a redteam \
  --agent wss://agent.deepgram.com/v1/agent/converse \
  --intensity advanced \
  --spec "Deepgram Voice Agent for billing and account support"

# Target specific attack categories only
testmu-a2a redteam \
  --agent wss://agent.deepgram.com/v1/agent/converse \
  --categories prompt-injection,jailbreak,pii-leakage

The output is a letter grade, A+ to F, per category instead of one pass or fail. That matters because an agent can resist a jailbreak attempt perfectly on the happy path and still leak PII through a function call nobody thought to probe. Run it before launch, then again after every prompt or model change that touches what the think layer is told never to do.

Wiring Regression Testing Into CI/CD

A think-layer prompt edit, or a Nova or Aura model upgrade Deepgram ships on its own side, can change conversation quality without a line of your code changing. Nothing catches that unless the pipeline does. testmu-a2a-cli installs with pip, authenticates from CI secrets instead of an interactive login, and writes JUnit XML that GitHub Actions, GitLab CI, and Jenkins already read natively.

# .github/workflows/deepgram-agent-tests.yml
name: Deepgram Agent Tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install TestMu A2A CLI
        run: pip install testmu-a2a-cli
      - name: Run Deepgram agent regression suite
        env:
          TESTMU_USERNAME: ${{ secrets.TESTMU_USERNAME }}
          TESTMU_ACCESS_KEY: ${{ secrets.TESTMU_ACCESS_KEY }}
        run: |
          testmu-a2a test \
            --agent wss://agent.deepgram.com/v1/agent/converse \
            --spec "Deepgram Voice Agent for billing and account support" \
            --format junit --output results.xml
      - name: Publish results
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: Deepgram Agent Test Results
          path: results.xml
          reporter: java-junit

Gate on exit code: 0 for all scenarios passed, 1 for any failure. Reserve the full run for think-layer prompt or model changes. A change that only touches the telephony bridge can run a smaller smoke set instead of the whole scenario library.

What Automated Testing Still Won't Catch

A Green verdict means the agent passed on the scenarios you ran, with the personas you tested, on the metrics you configured. It does not cover the following.

• Low-confidence Green isn't a cleared launch: it just means too few scenarios ran to trust the score yet. Run more, especially on hallucination and compliance, before you ship.
• Cross-session memory isn't evaluated: agents get tested per call, not across calls. If your agent is supposed to remember a caller from a previous session, test that separately.
• The concurrency numbers above test the platform's own throughput, not Deepgram's infrastructure under a real traffic spike. Pair evaluation with actual launch-day monitoring.
• Highly specialized domains need custom rules: the default 9 metrics cover general conversation quality well. An agent giving medical or legal guidance needs more than that.
• A stale agent prompt gives you a stale verdict: if the think layer's behavior changed and the evaluation prompt describing "correct" didn't, some regressions slip through and some good new behavior gets flagged as a failure.

None of this is a reason to skip testing. It's a reason to keep testing after launch, not just before it.

Conclusion

A Nova-3 benchmark score tells you how well the model transcribes clean audio. It doesn't tell you if the agent built on top of it can hold a conversation. Connect your Deepgram Voice Agent to a testing loop that scores task success, conversation quality, safety, and resilience across personas and accents the benchmark never saw. TestMu AI's Agent Testing does this end to end, from scenario generation through a production-readiness verdict, whether your agent runs over the raw WebSocket API or a phone number.

The fastest way to see it against your own agent is to connect its endpoint and generate scenarios from the same prompt or documentation you used to build it, the same first step the Testing Your First AI Agent docs walk through.