AI Agent Testing: Manual vs LLM-as-a-Judge vs Simulation

Your agent passed every scenario your team wrote, and it still shipped a bug that a customer found in four turns. Running the same checks harder would not have caught it. Most teams pick one way to evaluate an agent, and each of the three available methods is blind to a different class of failure.

This article compares the three on what actually decides the choice: what each costs, how much ground each covers, and how many real defects each one catches. The third method, simulation, gets the most space, because it is the one most teams have not built yet and the one that produces the conversations the other two grade.

Why a Single Testing Method Misses Agent Failures

Traditional automation asserts that a known input yields a known output. An agent returns different wording every run, so a strict equality assertion is flaky the moment you write it. The deeper problem is that correctness is not in any single response.

An agent can answer all five turns correctly in isolation and still fail the conversation, by forgetting the account number the user gave in turn one, or by refusing to escalate after the third failed attempt. These failures live in the relationship between turns, which is where single-response scoring has nothing to look at.

The measurement gap is quantified. In a study of a deployed multi-turn food-and-beverage ordering agent, Zhang, Wang, and Lei found the agent's built-in LLM judge surfaced 2 of 9 human-confirmed systematic problem patterns in one batch, and its operational gate flagged zero of 100 rounds in a batch where exhaustive human review confirmed 23 distinct defects and 7 new cross-cutting patterns. The judge was not broken. It was measuring agreement with ratings while the defects sat outside what it had been told to look for.

That finding reframes the whole methodology question. Judge reliability is almost always reported as agreement with human ratings, and recall of real defects is a separate number that is rarely published. Picking a method means deciding which blind spot you can live with.

• Manual review is bounded by how many transcripts a person can read, which is a few hundred a week at best.
• LLM judging is bounded by the rubric, so it scores the failure modes you already knew about and stays quiet on the rest.
• Simulation is bounded by the scenarios and personas you generate, so a narrow scenario set produces a narrow verdict no matter how many runs you execute.

None of the three dissolves the others. The useful question is which one owns which job, and that starts with seeing them side by side. For the dimensions each method should be scoring against, our breakdown of the AI agent evaluation framework covers task success, conversation quality, safety, and resilience.

The Three AI Agent Testing Methods Compared

One distinction resolves most of the confusion between these methods. Simulation produces conversations; manual review and LLM judging are two ways to grade conversations. Teams that treat all three as alternatives end up choosing between a generator and two graders.

Read the table column by column and the division of labour is clear. Simulation buys coverage, the judge buys throughput, and manual review buys the ground truth that tells you whether the other two are measuring anything real.

Manual Transcript Review: Slow, Small, and Still Necessary

Manual review means a person opens a full conversation and decides what went wrong in it. It is the slowest method per transcript and the only one that finds failure patterns nobody wrote a rule for.

The multi-turn ordering agent study makes the case concretely: exhaustive human review is what established that 23 defects and 7 cross-cutting patterns existed at all. Every one of those patterns was invisible to the automated gate until a human named it.

What manual review is good for:

• Discovering failure classes that do not yet exist in any rubric, which is how new rubric criteria get written in the first place.
• Calibrating a judge, by scoring the same sample both ways and comparing where the two disagree.
• Adjudicating genuinely subjective calls, such as whether a technically correct reply was appropriate for an upset customer.
• Investigating a specific incident, where reading ten real transcripts beats any aggregate score.

Where it breaks down is volume and consistency. Reviewers disagree about whether a response was complete or biased, and that disagreement is why a scored rubric exists at all. Treat manual review as a sampling instrument, not a gate: a fixed weekly sample, read closely, with findings written back into the rubric.

One practical rule keeps the sample honest. Draw it from failed and low-confidence runs rather than random ones, because a random sample of a mostly-passing suite spends most of the reviewer's hour confirming things already known to work.

LLM-as-a-Judge: Cheap Throughput With a Recall Ceiling

An LLM judge scores output against criteria you write, instead of a person reading and deciding. It is the cheapest way to put a number on every scenario in a suite, and it is the method most teams reach for second.

Two numbers get confused here, and the distinction decides how much weight the judge can carry. Agreement measures how often the judge and a human give the same verdict on the same item. Recall measures how many genuine defects the judge surfaces out of all defects present. The ordering-agent study reported the second number and found it far below what agreement scores imply, with 22% of confirmed patterns caught in one batch.

Use the judge where its economics are unbeatable and its ceiling does not matter:

• Scoring criteria that need semantic understanding, such as factual grounding, tone, or whether an answer was complete.
• Regression checks, where the question is whether a known metric moved rather than whether something new broke.
• Volume no human will ever read, where an imperfect score on every run beats a perfect score on 1% of runs.

Route anything with an objectively correct answer to plain code instead. Schema validation, required-field checks, character limits, and status codes are all solved by a line of Python, and sending them to a judge model pays tokens for a worse answer. Our practical guide to LLM-as-a-judge covers the scoring techniques and the biases to code against in depth.

The judge's ceiling is structural, so no amount of prompt work removes it. A rubric is a list of things you already know to worry about, which leaves the unknown failure modes to manual review and simulation.

Simulation: Generating the Conversations Worth Grading

Simulation drives the agent through complete conversations using synthetic users, through the same chat, voice, or phone interface production users hit. The output is a transcript that did not exist before, which is why simulation is a generator rather than a third grader.

This matters because the failures that cost the most only appear under conditions a scripted test never creates. A user who changes their mind in turn four, arrives already angry, speaks with an accent the speech model was not tuned for, or asks something outside the agent's scope will each expose a different weakness.

What a Simulation Run Actually Varies

A simulation suite is defined by three axes, and coverage comes from combining them rather than from raw scenario count.

• Scenarios describe what the user is trying to do, spanning happy paths, edge cases, adversarial inputs, and compliance situations.
• Personas describe how they behave while doing it, which is what separates an impatient caller from a confused one working the same task.
• Conditions describe the environment, and for voice agents that means accents, background noise, and connection quality that change what the agent even hears.

TestMu AI Agent Testing implements all three axes on one platform. It auto-generates 60 to 100+ scenarios from an uploaded PRD, knowledge base, or Jira ticket, ships 10 pre-built persona types including Impatient User, Confused Customer, Angry User, and Off-Script User, and for voice and phone agents adds 200+ voice profiles across 50+ accents with 15 background noise presets. Custom personas are defined by communication style, vocabulary level, emotional state, and interaction patterns.

The screenshot above was captured from the live Agent Testing product page and shows the four controls that define a simulation run: autonomous scenario generation, reusable test profiles and personas, evidence-based validation criteria with High, Medium, and Low confidence tracking, and the scheduling engine that reruns suites on a cron expression.

Why Repeat Runs Are the Whole Point

Running a scenario once tells you whether the agent passed that time. Because the system is non-deterministic, the same scenario can pass and then fail on the next attempt with no change to the agent.

Report how often a scenario passed rather than whether it passed, and the deployment conversation changes. A scenario that passes nine times in ten is a different risk from one that passes five times in ten, and a single-run suite reports both as green.

Confidence tracking makes this legible. Agent Testing attaches a High, Medium, or Low confidence level to every metric score based on how many scenarios backed it, so a passing metric with Low confidence reads as "run more scenarios" rather than "cleared for launch". The run then rolls up into a Green, Yellow, or Red production-readiness verdict instead of a raw score a reader has to interpret.

What Simulation Still Cannot Do

Simulation is bounded by the scenarios and personas you generate. A narrow configuration produces a narrow verdict, and a Green result means the agent passed the scenarios tested with the personas configured. Production can still surprise you.

It also costs more per conversation than scoring an existing transcript, because every run drives the live agent and incurs whatever that agent costs to operate. That economics is why full simulation belongs on releases rather than on every commit.

Cost, Coverage, and Recall Side by Side

Method comparisons usually stop at capability lists. The three numbers that decide a real budget are what a method costs as you scale it, how much of the failure space it can reach, and how many genuine defects it returns.

The recall column is the one worth arguing about internally. A judge that agrees with your reviewers 80% of the time sounds like a reviewer replacement, and the ordering-agent numbers show it can still miss most of what those reviewers find, because agreement is measured only on items someone already flagged.

Cost behaves differently across the three in a way that shapes scheduling. Manual review is expensive per item and cannot be scaled by spending more on infrastructure, judging is cheap per item but is paid on every single run, and simulation carries a per-conversation cost that makes frequency the lever you actually control.

How to Choose: Decision Rules by Failure Class

"Use a hybrid approach" is where most guidance stops, and it leaves the reader to work out the split. These rules map a failure class to the method that actually detects it, so the split follows from your risk rather than from budget alone.

Two rules cut across the table. Anything with an objectively correct answer goes to deterministic code before it reaches a judge, and any metric you plan to gate a release on gets a manual calibration sample before you trust it.

Agent surface changes the weighting too. A text chatbot can lean harder on judging because its transcripts are clean, while a phone agent needs simulation to carry more of the load, since accent, noise, and interruption handling have no textual equivalent to score. Our guide to conversational AI testing works through the chatbot and voice-agent differences.

Running All Three Methods in One Pipeline

The three methods run at different frequencies, and that scheduling is what makes the combination affordable. Simulation and judging belong in CI; manual review runs on a human calendar beside it.

• On every commit, run a reduced scenario set and gate on the exit code, so a broken prompt fails the build in minutes rather than in production.
• On agent prompt changes and model updates, run the full simulation suite, because these are the changes most likely to move behaviour in ways nobody predicted.
• On a schedule, rerun the suite against an unchanged agent to catch upstream model drift and knowledge-base staleness.
• Weekly, pull a manual sample from failed and low-confidence runs and write what you find back into the rubric and the scenario library.

TestMu AI ships this as a CLI, so the pipeline wiring is standard. The testmu-a2a-cli installs with pip, authenticates from environment variables, emits JUnit XML that CI platforms report natively, and returns exit code 0 when all tests pass and 1 on failure.

# Reduced scenario set on every commit
- name: Agent quality gate
  env:
    TESTMU_USERNAME: ${{ secrets.TESTMU_USERNAME }}
    TESTMU_ACCESS_KEY: ${{ secrets.TESTMU_ACCESS_KEY }}
  run: |
    pip install testmu-a2a-cli
    testmu-a2a test \
      --agent ${{ vars.AGENT_ENDPOINT }} \
      --spec "Customer support agent for billing and account issues" \
      --count 10 \
      --format junit \
      --output results.xml

Raise --count for the full release suite and keep the reduced set for per-commit runs, which is the frequency lever that keeps simulation cost proportional to risk. Agents behind a corporate firewall do not need a public endpoint: execution runs through HyperExecute with a secure tunnel to the private agent.

Keep the results file as an artifact on every run. It is the audit trail that lets you answer which scenarios passed at which agent version, and it is the evidence a compliance reviewer will ask for.

Where to Start This Week

Take twenty transcripts from your agent's real traffic, read them yourself, and write down every distinct thing that went wrong. That list is your first rubric, and it will contain failure classes your current tests do not check for.

Then close the loop in order. Turn the list into scored criteria, generate simulation scenarios that reproduce each failure with a persona likely to trigger it, and run each scenario several times so the report tells you how often it passes rather than whether it passed once.

To run that loop without building the harness yourself, connect your agent to TestMu AI Agent Testing: upload a PRD or knowledge base, add the behavioural prompt, and the platform generates the scenario set, executes it through personas against your live endpoint, and returns a Green, Yellow, or Red verdict with the failing transcripts annotated. The testing your first AI agent guide walks through the first run, and the AI agent evaluation tools roundup covers how the wider tooling category compares.

Everything above scores what the agent said. When the agent acts instead of answers, and the risk is that the effect does not match the account, Agent Assurance grades the filesystem deltas and tool calls behind the reply and reports what it could not verify.