Agent Testing CLI: CLI Based Testing for AI Agents

What Is CLI Based Testing?

CLI based testing runs a command-line tool the way a user runs it, then asserts on three outputs: the exit code, stdout, and stderr. ISTQB defines it as submitting commands to the software under test.

The scope is wider than correctness alone. Command-line interface testing verifies the functionality, usability, and performance of an application through its command-line interface, so that commands, arguments, options, and outputs all work as expected.^[2]

The distinction that matters is black-box versus unit. A unit test imports a function out of the tool and calls it directly, which proves the function works but not that the tool does. A black-box test spawns the built binary as a subprocess, so it exercises argument parsing, config loading, environment handling, and output formatting in one pass.

Georgy Marchuk argues the same case in Smashing Magazine. He borrows the Testing Library principle that the more your tests resemble the way your software is used, the more confidence they give you.^[3] For a CLI, resembling real use means a shell command, not an imported module.

Compared with GUI testing, the surface is much smaller. A GUI test needs a browser or device, a locator strategy, and a wait strategy. A terminal has no rendering layer to race against, so command-line tests run faster and stay far less flaky.

What Does a Command-Line Test Suite Check?

A command-line test suite checks the exit code, stdout, stderr, and the reply to stdin. Exit code 0 means success, stdout is the API scripts parse, and stderr carries errors. Bats and pytest run them.

• Exit code: Assert the exact number, not just non-zero, because a tool that distinguishes 1 from 2 is telling a CI script which failure occurred.
• Stdout contract: Treat machine-readable output as an API and assert against a stable format such as JSON, since users will parse whatever the tool prints.
• Stderr separation: Confirm errors go to stderr and results go to stdout, because a tool that mixes them corrupts every downstream pipe.
• Interactive prompts: Write to stdin when a prompt appears and assert on the reply, which is the only way to cover a wizard or a confirmation step.
• Environment isolation: Run each case in a clean working directory with a controlled environment so a leftover config file cannot make a broken test pass.

Pick a runner that already handles subprocess plumbing rather than building one. Bats is a TAP-compliant testing framework for Bash, and shelltestrunner reads plain-text specs of a command, its input, and its expected output.

Judo, an open-source framework from Intuit, is driven from a YAML file that declares which commands to run and how to assert the outcome. Judo can also respond to stdin when an expected stdout string appears, which is what makes an interactive prompt testable.^[4]

The CLI Testing Library offers Testing Library-style utilities for end-to-end CLI tests and lets you bring your own testing framework.^[5] Python teams generally reach for subprocess inside pytest, which already gives clean access to all three outputs. The same plumbing question comes up in MCP vs CLI when agent tooling has to be driven programmatically.

How Does AI Agent Testing Differ From CLI Based Testing?

The assertion changes. CLI based testing compares stdout to an exact string. An AI agent replies differently every run, so testmu-a2a-cli scores quality dimensions against a threshold instead.

The replacement is a scored assertion. Instead of checking that the agent said a specific sentence, the evaluation scores the response across quality dimensions and compares those scores to a threshold. A hallucination score below the configured bar fails the run; the exact wording never enters the assertion.

Coverage stops being a fixed list of cases and becomes a generated population of scenarios, because no engineer writes 200 adversarial prompts by hand. And a single run is no longer proof, since a non-deterministic system can pass once and fail the next time, which is why scenario count and a pass threshold both matter more than any individual result.

The shape of the test stays familiar. The command still runs as a subprocess, still writes structured output, and still exits non-zero when the run falls below the bar, so it drops into the same pipeline step as any other command-line test.

What Is the Agent Testing CLI (testmu-a2a-cli)?

The Agent Testing CLI is testmu-a2a-cli, a Python package that runs Agent Testing evaluations from your terminal. It scores chat, voice, and phone agents across nine quality dimensions.

TestMu AI's Agent Testing platform now has a CLI, and today we're extending it to the command line. Here's what that means for your workflow.

AI agents are in production. Chatbots handle customer queries at scale. Voice assistants route support tickets. Calling agents close loops without a human in the loop, and most QA teams are still testing these systems by hand, one conversation at a time.

That bottleneck is what TestMu AI built Agent Testing to fix, and the command line is now a first-class way to reach it. Here's what a full evaluation run looks like from your terminal:

# Install
pip install testmu-a2a-cli

# Authenticate
testmu-a2a auth --username YOUR_USERNAME --access-key YOUR_KEY

# Run a quick evaluation against your agent
testmu-a2a test \
  --agent https://your-chatbot-endpoint.com \
  --spec "E-commerce customer support chatbot" \
  --count 200 \
  --format json \
  --output results.json

The nine chat and voice quality dimensions scored out of the box are hallucination detection, bias detection, completeness, context awareness, response quality, conversation flow, tone consistency, positive user outcome, and root-cause understanding.

How to Evaluate AI Agents at Scale From the Terminal?

Run testmu-a2a test with your agent endpoint, a plain-language spec, and a scenario count. Autonomous evaluators run hundreds of adversarial conversations in parallel and return scored JSON.

Let's take this into practice.

Use case: Your team has built a customer support chatbot for an e-commerce platform. It handles order queries, refund requests, and product FAQs.

You're three days from shipping it to production and you need to know: does it hallucinate? Does it stay on topic when users go adversarial? Does it handle edge cases around refund policy correctly?

The old way: A QA engineer writes 30-50 test scripts manually, runs them one by one, and files bugs based on what they notice. It takes days. It misses edge cases because humans don't think adversarially at scale.

With testmu-a2a-cli: Point the CLI at your agent's endpoint, define your spec, and let autonomous evaluators run hundreds of realistic and adversarial scenarios against it in parallel. In minutes, you have structured quality scores across Hallucination Detection, Bias Detection, Response Quality, Conversation Flow, and more. No scripts, no manual review.

The customer support example is just one case. The same pattern applies anywhere you are shipping a conversational AI system:

• Chatbots: Need validation across hundreds of user intent paths before going live.
• Voice assistants: Handle phone calls and must stay coherent across multi-turn conversations.
• Calling agents: Book appointments or handle escalations, where one wrong response has real consequences.
• Internal AI tools: Interact with employees and must stay within compliance boundaries.
• Embedded LLM features: Ship inside products, where hallucinations and off-topic responses are silent failures.

Spin up your Agent Testing CLI in just minutes with this detailed documentation.

How to Red Team an AI Agent Before Production?

Run testmu-a2a redteam against your agent endpoint. It sends adversarial prompts across nine attack categories, from prompt injection to PII leakage, and reports which ones the agent failed to refuse.

One of the most important capabilities in testmu-a2a-cli and the one most teams don't think to use until it's too late is redteam. The redteam command measures whether a motivated user can break the agent.

testmu-a2a redteam \
  --agent https://your-chatbot-endpoint.com \
  --output redteam-results.json

The nine categories are prompt injection, jailbreak, data exfiltration, PII leakage, harmful content, overreliance, hijacking, policy violation, and technical injection. An --intensity flag sets how hard each category pushes, from basic through intermediate to advanced.

If your agent handles sensitive data, makes decisions with real-world consequences, or is customer-facing, red teaming before launch is not optional. The CLI makes it a single command.

Can You Test Voice and Phone Agents From the Command Line?

Yes. testmu-a2a call tests voice and phone agents from the command line, alongside chat testing. It places real inbound and outbound calls and scores each one against 30+ call metrics.

testmu-a2a call \
  --agent https://your-phone-agent-endpoint.com \
  --type inbound \
  --output call-results.json

The CLI supports 30+ phone-agent quality metrics, background sound simulation, and DTMF detection, covering the real conditions your voice agent will face in production.

Multi-turn coherence, intent handling under noise, escalation behavior, all testable from the terminal.

For a deeper look at the metrics the CLI scores against, including MOS, PESQ, POLQA, WER, endpointing latency, and the multi-turn failure modes that production reveals, see this complete guide to voice quality testing for VoIP and AI voice agents.

How to Make Agent Evaluations Repeatable With init and run?

Run testmu-a2a init to generate a testmu-a2a.yaml config, commit it beside your code, then run testmu-a2a run. Every environment executes the identical evaluation with no flags in the pipeline.

For teams who want repeatable, version-controlled evaluation runs, testmu-a2a init generates a testmu-a2a.yaml config file that you can commit alongside your codebase and run consistently across environments:

# Generate your config file
testmu-a2a init

# Run from config - works in CI/CD without any flags
testmu-a2a run --config testmu-a2a.yaml

This is how most teams use the CLI in practice: testmu-a2a test for fast ad-hoc checks during development, and testmu-a2a run with a committed config for pipeline-controlled evaluation gates.

Committing the config also puts your quality bar under review. A threshold change becomes a pull request that a reviewer can see, instead of a setting somebody edited in a console.

How to Verify Browser Flows From the Same Terminal?

Run Kane CLI headless. It takes a natural-language objective, drives a real Chrome browser, prints JSON on stdout, and exits non-zero on failure, the same contract testmu-a2a-cli uses.

testmu-a2a-cli scores conversations rather than rendered pages. That gap is normally covered by a browser test framework, and the cost is selector maintenance. A button gets a new CSS class, a label is reworded, a layout shifts, and the test fails on a change no user would notice. Teams end up maintaining locators instead of checking behavior.

Kane CLI takes a different input. You write the objective in natural language, it drives a real Chrome browser, and it returns a pass or fail with test run evidence attached. Its headless mode is built for exactly the pipeline step this article has been describing:

kane-cli run "Verify the homepage loads and the signup form submits" --headless

Three behaviors matter when a browser check runs unattended:

• Vision-based waiting: The agent watches the rendered viewport rather than network idle or DOM ready, so canvas elements, shadow DOM, and streamed renders do not produce false failures.
• Autoheal with confidence scoring: The objective is anchored to the user-facing element, so a reworded label re-resolves instead of failing, and a low-confidence match is rejected rather than passed silently.
• Step ceiling: A run pushes through cookie banners and redirects up to 50 steps by default, tunable with --max-steps, so long journeys are split deliberately rather than dying halfway.

The limits are worth stating. A vision-driven run costs more wall-clock time than a selector assertion, and a deliberately strict regression test is still better served by an exact selector. The contract, though, is the same one every command-line test uses: run it, read the exit code, parse stdout.

Why Is the Agent Testing CLI Important for Your Pipeline?

The Agent Testing CLI is important because a terminal command can fail a build and a browser console cannot. testmu-a2a-cli turns a failing bias or hallucination score into a build-blocking exit.

The Agent Testing platform has always had a clear thesis: you can't use deterministic, script-based QA to validate non-deterministic AI systems. Static test cases don't adapt, they miss edge cases.

Agent Testing answers that thesis by deploying autonomous evaluators that emulate real users and intelligent adversarial interactions. Until now, reaching that capability meant going through the TestMu AI browser-based console. The CLI changes that for teams who live in the terminal, run tests in CI/CD pipelines, and want evaluation results feeding directly into deployment gates.

Drop testmu-a2a test into your GitHub integration or a Jenkins step. Set thresholds. Fail the build if hallucination or bias scores miss your bar. Results come back as structured JSON, parseable, alertable, and ready to feed into any dashboard.

CLI based testing for AI agents gives an agent team the same deployment gate that unit tests already give a service team. Start by running one evaluation against a staging endpoint and reading the scores before you set a threshold, because a bar chosen without a baseline either blocks every deploy or none of them.

A general-purpose agent can drive a browser too, but it optimizes for finishing the task rather than proving it works. Kane CLI vs Manus walks through where that difference shows up in QA work.

This CLI grades what an agent says. For agents that act instead, writing files, calling APIs, and issuing refunds, Agent Assurance grades the effect against observed evidence and reports the criteria it could not verify.

Sources and References Used

• [1] CLI Testing - ISTQB Glossary, Advanced Test Automation Engineer Syllabus (2016)
• [2] What Is Command-Line Interface Testing? - Onion Training Glossary
• [3] Testing The CLI The Way People Use It - Georgy Marchuk, Smashing Magazine (April 2022)
• [4] Judo: CLI Integration Testing Framework - Intuit, GitHub
• [5] CLI Testing Library Documentation - cli-testing.com