How to Test a LiveKit Agent

What Is LiveKit and What Are LiveKit Agents?

LiveKit is both a WebRTC infrastructure project and a framework for building AI agents on top of it. The core livekit/livekit repository is a distributed WebRTC SFU (selective forwarding unit), the media server that moves audio, video, and data between participants in real time. Its data model has three parts: a room is the virtual space where a session happens, participants are the users, agents, or services in that room, and tracks are the audio, video, or data streams each participant publishes or subscribes to.

LiveKit Agents is a separate, higher-level framework that runs as a server-side participant inside a room. Its core class, AgentSession, combines a speech-to-text provider, an LLM, a text-to-speech provider, and turn-detection handling into one streaming voice pipeline. A minimal agent looks like this:

from livekit import agents
from livekit.agents import AgentSession, Agent, RoomInputOptions
from livekit.plugins import deepgram, openai, cartesia, silero

class SupportAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="You are a customer support agent for Acme Corp. "
                         "Be concise, empathetic, and never invent account details."
        )

async def entrypoint(ctx: agents.JobContext):
    session = AgentSession(
        vad=silero.VAD.load(),
        stt=deepgram.STT(model="nova-3"),
        llm=openai.LLM(model="gpt-4o"),
        tts=cartesia.TTS(),
    )

    await session.start(
        room=ctx.room,
        agent=SupportAgent(),
        room_input_options=RoomInputOptions(),
    )

if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))

Phone integration works through LiveKit SIP, generally available since 2025. Inbound calls arrive through a SIP trunk and get routed into a room by dispatch rules; outbound calls are placed through LiveKit's server API. Either way, callers show up as ordinary SIP participants, so the agent interacts with them the same way it would with any other room participant. Full detail lives in the telephony integration docs.

LiveKit offers both a fully self-hosted path and a managed LiveKit Cloud tier. Cloud pricing starts with a free Build tier (1,000 agent session minutes, 5 concurrent sessions), a Ship tier from $50 a month (5,000 minutes, 20 concurrent sessions), and a Scale tier from $500 a month (50,000 minutes, up to 600 concurrent sessions), with custom Enterprise plans above that. Self-hosting the open source media server and Agents framework carries no LiveKit licensing cost, only your own infrastructure.

How Do LiveKit Agents Fail in Production?

Because a LiveKit agent is assembled from whichever plugins the developer picked, most production issues trace back to that specific combination rather than a bug in LiveKit itself.

• Turn detection tuned too sensitively interrupts users mid-sentence, and developers have reported not being able to tune "thinking" versus "speaking" states independently (livekit/agents issue #3427).
• Latency that looks fine locally can double once a call routes through telephony, since the STT-to-LLM-to-TTS chain has to complete before audio reaches the phone leg (issue #3685).
• Some STT providers show a multi-second cold-start delay on the first request of a session before settling into normal response time, which a single warmed-up test call will not catch (issue #1893).
• SIP audio quality issues, like fading or grainy words on outbound calls, surface only under real telephony conditions, not in a browser-based test session.

None of this is a defect in LiveKit's design. A framework leaves component choice to the developer, and the tradeoff is that validating those choices together, not just individually, is also the developer's job.

What Should You Test in a LiveKit Agent?

Every voice agent, regardless of framework, needs coverage across four dimensions: task success, conversation quality, safety, and resilience. For a LiveKit agent specifically, resilience testing has to account for the exact plugin combination in that deployment.

These four dimensions are the same ones covered in the broader AI agent testing framework, applied here to a LiveKit-specific pipeline.

What Does LiveKit's Built-In Testing Cover, and Where Does It Stop?

Unlike some voice frameworks that ship no evaluation layer at all, LiveKit provides a first-party testing module built as a thin layer over pytest (Python) and Vitest (Node.js).

• A test-mode AgentSession and a RunResult object capture the events an agent produces during a scripted exchange, so you can assert on what happened.
• A judge(llm, intent="...") assertion hands a response to an LLM and asks whether it matched a stated intent, which is a form of LLM-as-a-judge scoring applied to individual turns.
• Testing runs in text mode by default, using an LLM through LiveKit Inference or a plugin in text-only mode, which keeps tests fast and cheap but skips the STT and TTS stages entirely.

The same documentation is direct about what this does not cover: end-to-end audio-pipeline testing, and production monitoring, both of which it points developers to third-party tools for. In practice, that means the built-in layer is well suited to catching a broken tool call or a clearly wrong response during development, and not suited to catching an accent your STT mishandles, a turn-detection setting that misfires under background noise, or a tone that drifts across a real ten-turn phone call.

That gap, between unit-level text assertions and full-conversation, real-audio, multi-persona testing, is exactly where a dedicated evaluation platform earns its place alongside LiveKit's own test layer rather than instead of it.

How Do You Test a LiveKit Agent, Step by Step?

Testing a deployed LiveKit agent means triggering real conversations against the room or phone number it answers on, capturing what happened, and scoring it against the four dimensions above.

• Run LiveKit's own pytest or Vitest suite first, to catch broken tool calls, missing instructions, or obviously wrong responses before the agent ever reaches a real room.
• Deploy the AgentSession worker you want to test, reachable through a room, a SIP trunk, or a tunnel for local iteration.
• Trigger real or synthetic conversations against that deployment, the same way an actual caller or user would connect.
• Capture full transcripts, and audio where latency and interruption handling matter, not just the final text response.
• Score the conversation against task success, quality, safety, and resilience, since the same input can legitimately produce different phrasing on different runs.
• Repeat across personas, accents, and noise conditions, not just the one clean call a developer runs by hand.

Steps three through six are where TestMu AI's Agent Testing does the work. It treats your LiveKit agent as a black box reached through the room or phone number its AgentSession answers on: you give it the agent's role and policies once, and it generates the synthetic callers, connects to the deployment, runs each caller through a full multi-turn conversation, and grades what comes back, whichever STT, LLM, and TTS plugins sit behind it.

The Agent Testing CLI, TestMu AI's testmu-a2a-cli, drives that from the terminal, calling the deployed endpoint the way a real caller would:

# Call the room or phone number your LiveKit agent answers on
testmu-a2a call \
  --number +15551234567 \
  --persona confused \
  --scenario "Customer wants to reschedule an appointment" \
  --voice Neha

# Run a full regression suite against the same deployment
testmu-a2a suites run --project <project_id> --name "Regression"

Each conversation is scored on 9 conversation-quality metrics, including hallucination, completeness, and context awareness, plus 30+ phone-specific metrics like first call resolution, containment rate, and DTMF handling accuracy. The run rolls up into a Green, Yellow, or Red readiness verdict, with the failing transcripts attached so you can see the turn that broke instead of guessing at it. When a self-hosted AgentSession runs inside private infrastructure, the platform reaches it through HyperExecute's secure tunnel, so the room or SIP endpoint under test never needs a public address.

How Do You Keep a LiveKit Agent Reliable After Launch?

The run above gets an agent ready for its first real call. Keeping it ready is a separate job, because a LiveKit agent's behavior drifts even when the room, dispatch rules, and pipeline wiring never change.

• A prompt edit, a swapped STT or TTS plugin, or an LLM version bump can shift how the agent answers with nothing failing in LiveKit's own worker logs, so a scored baseline is what surfaces the regression before a caller runs into it.
• Scheduled runs re-score the same suite on a daily or per-deploy cadence, catching drift from upstream model updates in the stretches between your own releases, the same drift LiveKit's text-mode judge assertion was never meant to track.
• Every failure a real caller finds becomes a permanent test case, so the persona library grows into a record of everything the agent has been taught to handle rather than a one-time checklist.
• Running the calls in parallel keeps that regression suite fast enough to gate a pull request, so voice quality stays a release check instead of a post-incident cleanup.

This ongoing discipline is the subject of voice agent regression testing in depth, and it pairs with the four-dimension approach covered in AI agent evaluation beyond pass/fail.

Conclusion

LiveKit gives developers real infrastructure and a real, if partial, testing layer, which puts it ahead of frameworks that ship no evaluation tooling at all. The part it leaves open is exactly the part that matters most once an agent is live: whether it holds up across real audio, real accents, and a real adversarial caller.

Start with the room or phone number you already have deployed, not a rewrite of your pipeline. Run LiveKit's own test suite for the checks it does well, then capture a baseline set of full-conversation test calls, including at least one adversarial persona, so the next plugin swap has something concrete to regress against. Follow the testing your first AI agent guide to connect your build and run the first suite.