7 Botium Alternatives for LLM Chatbot and Voice Testing

1. TestMu AI Agent Testing (Formerly LambdaTest)

TestMu AI publishes this guide, and this entry is its own product. Its AI agent testing platform works with chat, voice, phone, video and image agents. Chat tests reach your bot's endpoint over standard HTTP in your API's own request and response shape, even on a private network or localhost through TestMu AI's secure proxy, and an endpoint profile can be imported from a Postman collection.

It is the one tool here that passes the platform-fit test: the chatbot testing platform page links testing pages for Dialogflow, Amazon Lex, watsonx Assistant, Rasa, Botpress, Voiceflow, Kore.ai and Cognigy, and the docs include setup guides for testing Amazon Lex bots and watsonx Assistant bots with TestMu AI.

From your prompt and documents (PDF or text uploads, or Confluence, JIRA and GitHub as sources), the platform builds 60 to 100+ scenarios and plays each one as a multi-turn conversation with pre-built or custom personas. Each conversation is scored on 9 quality metrics, Bias Detection, Hallucination Detection and Response Quality among them, and each scenario can carry validation criteria that return Pass, Fail or Unable to Verify. A Go-Live Assessment then rates the agent Green, Yellow or Red.

Phone tests place real calls with 200+ voices and 15 background-noise presets, detect the keypad tones in each call and show them in the speaker-identified transcript, and score every call on 30+ metrics, including first call resolution and containment rate. Video tests send a simulated candidate to your agent's web URL and judge it against criteria you write, and image tests give generated images a 0 to 100 quality score. From CI, the Agent Testing CLI starts a suite; a chat suite also needs its workflow ID:

pip install agent-testing-cli
agent-testing-cli --project PROJECT_ID run \
  --workflow WORKFLOW_ID --suite SUITE_ID --yes

In a pipeline, the CLI takes the account name from LT_USERNAME and the key from LT_ACCESS_KEY. Chat evaluations run asynchronously and are not started with the wait flag, so a pipeline collects their results in a later step; a Phone Caller run can wait for its result, and a failed test in that run returns exit code 1.

• Best for - Botium suites that test Dialogflow, Lex, watsonx Assistant or Rasa bots, Botium Box users moving to a hosted platform, and suites that must grow from chat into voice, phone, video or image agents.
• Limits - it does not import BotiumScript, so convos are rewritten as scenarios; suites live in a TestMu AI project rather than your repository; AI generation for phone agents stops at 20 inbound or 7 outbound scenarios; and video testing is switched on per organization.

2. LangWatch Scenario

LangWatch Scenario is an Apache-2.0 framework for testing agents, available in Python, TypeScript and Go (the Go SDK sits in a separate repository). You describe a scenario rather than a script: a user simulator agent sends messages until the scenario succeeds or hits its turn limit, and a judge agent checks your criteria at every turn and can end the run with a verdict, as the project's README describes.

The README treats voice as first-class, listing shipped adapters for ElevenLabs, OpenAI Realtime, Twilio Media Streams, Pipecat WebSocket and Gemini Live, with helpers for background noise and interruptions. Its Twilio adapter connects to a phone number and drives a full inbound or outbound call with DTMF detection, and a RedTeamAgent covers adversarial runs.

• Best for - teams whose botium-core suites were code in a repository and whose agent now also handles voice or phone traffic.
• Limits - scenarios, criteria and CI wiring are code you maintain; no Dialogflow, Amazon Lex, Watson Assistant or Rasa connector ships with it, and LiveKit, Vapi and generic WebRTC adapters are listed as follow-up work.

3. Coval

Coval's homepage carries the headline "Scale voice AI agents with confidence", and its documentation describes a deployment-readiness platform that tests chat agents as well as voice agents. The product splits into Simulate (thousands of realistic callers, hard policies and noisy audio), Observe (production evals on live conversations) and Human Review, which routes the calls that matter to human reviewers.

Its simulated user calls an agent's phone number or SIP address, and a chat agent connects through a standard HTTPS endpoint whose request template and response path you configure, or over WebSocket, A2A or SMS. The FAQ describes repeatable voice AI regression testing across prompt changes, model updates and vendor swaps, and the docs add guides for IVR routing, accents and red-team testing.

• Best for - voice and chat teams that want simulation, live-call evals and human review in one loop.
• Limits - runs launch against test sets stored in Coval, built with AI, by hand or from a CSV or Excel upload, and its standard chat endpoint must be a public HTTPS URL, since local and private IP addresses are not accepted.

4. Cekura

Cekura's site describes automated QA for voice AI and chat AI agents. It runs thousands of synthetic conversations before go-live, probes for jailbreaks, leaks and off-script behavior, detects drift across live calls, and replays the same scenarios across platforms and models; when it flags an issue, it reproduces the issue in simulation and suggests a fix.

Its Tests as Code guide puts up to 200 test cases, each with an optional expected outcome, in a cekura.tests.json file committed beside the agent, and the cekura CLI runs that file from CI and can fail the job unless every run passed. The docs list phone calls over the public telephone network or a SIP trunk, WebRTC sessions with LiveKit, Pipecat, Vapi, Retell and Agora agents, and chat through a provider's chat connection, your own WebSocket server, SMS, WhatsApp or an API Chat mode in which your code relays each turn.

• Best for - teams whose botium-core convos lived in a repository and who want hosted runs for chat, voice and phone agents, or a model comparison on the same scenarios.
• Limits - a bot that answers only over HTTP needs your own relay code or a WebSocket server, and Dialogflow, Lex and Watson, the platforms several Botium connectors target, are not among the integrations on its partners page. The Cekura alternative page compares it with TestMu AI.

5. Hamming

Hamming's homepage headline reads "The trust platform for voice and chat agents", and the product spans pre-launch testing, red teaming and production monitoring. You paste your agent's system prompt, scenarios are generated from it and scored by 50+ built-in metrics, and the tests cover accents, interruptions, background noise, DTMF and IVR trees.

Hamming dials a SIP number for inbound and outbound calls or connects to LiveKit and Pipecat agents over WebRTC, and it imports agents from Vapi, Retell, ElevenLabs or any orchestration framework. The red-team suite covers prompt injection, jailbreaks, PII leakage, policy violations and social engineering, and the homepage states that Hamming maintains SOC 2 Type II compliance and supports HIPAA.

• Best for - voice-agent teams on Vapi, Retell, LiveKit or Pipecat that want red teaming and live-traffic monitoring in one product.
• Limits - its public pages do not describe how a chat agent connects, and its documentation site asks for an access code; the homepage does not mention video agents or NLU intent testing. The Hamming AI alternative page compares it with TestMu AI.

6. Promptfoo

Promptfoo's README calls it a CLI and library for evaluating and red-teaming LLM-based apps. Tests sit in a configuration file, run with promptfoo eval and open in a web viewer with promptfoo view, while the HTTP or WebSocket provider sends them to a deployed bot's endpoint, the job Botium's generic HTTP/JSON connector does.

Multi-turn tests use the simulated user provider: its instructions template defines the simulated user, and its docs give a default of ten turns, which maxTurns changes. An ElevenLabs provider tests voice agents built on that platform. OpenAI announced on March 9, 2026 that it would acquire Promptfoo and keep building the open-source project, and the README now describes Promptfoo as part of OpenAI, still open source under the MIT license.

• Best for - teams that ran botium-cli against convo files and want the same shape: test files in the repository and one command in CI.
• Limits - no provider for Amazon Lex, Dialogflow, Rasa or telephony, so a bot on those platforms is tested through its HTTP endpoint, and none of the providers places phone calls.

7. DeepEval

DeepEval is an Apache-2.0 framework that its README describes as similar to Pytest but specialized for unit testing LLM apps. G-Eval scores custom criteria with an LLM judge, and the multi-turn metrics include Knowledge Retention, Conversation Completeness, Turn Relevancy, Turn Faithfulness and Role Adherence.

According to DeepEval's documentation, its conversation simulator plays a user against your chatbot from a golden that holds a scenario, an expected outcome and a persona, then returns conversational test cases for those metrics. You connect it through a model_callback you write, or in voice mode it speaks synthesized turns over a live connection and records audio and latency for each turn. Confident AI, built by the same founders, is the hosted platform for production monitoring.

• Best for - teams that ran Botium through its Mocha or Jest bindings and want conversation tests inside a Python test runner.
• Limits - the connection to your bot is code you write, and the simulator documentation describes no phone connector.

Steps to Move a BotiumScript Suite

• Inventory the suite - Collect the *.convo.txt and *.utterances.txt files and any scripting memory files, and tag each convo by business flow, such as orders, refunds or escalation.
• Keep exact checks as assertions - Buttons, links and card payloads have one right answer, so keep them as hard assertions or move them into API or UI tests you already run.
• Write a goal per convo - "Order a large pizza and get an order number" is a goal; the #me lines become what the simulated user wants rather than a script it follows.
• Turn utterance lists into personas - Instead of dozens of phrasings of one request, describe who sends them: terse, chatty, a non-native speaker, impatient or off-topic.
• Rewrite #bot steps as criteria - Each expected reply becomes a checkable statement, such as "confirms the size and toppings before placing the order", plus a must-not, such as "never invents a delivery time".
• Add the LLM-era checks - Cover grounding in your knowledge base, policy rules, prompt-injection attempts and the handling of personal data.
• Run both suites - Point the old and the new suite at the same build for a few releases, and retire a convo once its scenario catches the regressions the convo caught.
• Rebuild phone flows - When the agent also takes calls on a phone number, rebuild its key scenarios as phone scenarios with voices and background noise; phone calling agent testing covers inbound and outbound calls.

Example: A Pizza Order Convo Before and After

Before: a convo in the format the Botium documentation describes, where the #me line names an utterance list, BUTTON simulates a click, and $orderNum needs the SCRIPTING_ENABLE_MEMORY capability switched on:

Order pizza happy path

#me
USER_ORDER_PIZZA_UTT

#bot
INTENT I_ORDER_PIZZA
INTENT_CONFIDENCE 70
BUTTONS Small|Medium|Large

#me
BUTTON Large

#bot
Your order number is $orderNum

The utterance list behind USER_ORDER_PIZZA_UTT lives in a file whose name ends in .utterances.txt:

USER_ORDER_PIZZA_UTT
I want to order a pizza
can I get a pizza delivered
pizza please

After: the same flow as a scenario, written as an illustrative, generic spec that no tool in this list uses as its format:

# Illustrative scenario spec, not any tool's format
scenario: Order a large pizza
goal: Order a large pizza and get an order number
persona: Busy parent who answers in a few words
  and changes the topping halfway through
criteria:
  - Offers the size options before confirming
  - Reflects the topping change in the summary
  - Gives an order number in the final turn
must_not:
  - Invent a delivery time the order API did not send
  - Ask for card details in the chat
hard_checks:
  - Buttons shown: Small, Medium, Large
  - Order number exists in the order API
repeat: 10 runs, report the pass rate

The convo fails when the agent words the reply differently or the bot loses its intent layer. The scenario is written to pass any correct wording and fail a wrong one, including an invented delivery time, which a pattern list catches only if someone predicted the wording.