AI Agent Simulation: Simulated Users, Personas and Scoring

Four Meanings of Agent Simulation

Agent simulation can mean any of the practices in the table, and search results mix them freely. Only the first is a way to test an agent before release, and it is the subject of the rest of this guide.

Benchmarks sit between the first two rows: tau-bench, published by Yao and colleagues in June 2024, pairs a simulated world of tools and policy rules with a language model that plays the user. It is built to compare agents on shared tasks.

The third meaning is research on people, and it now produces measurable results. Park and colleagues (arXiv 2411.10109, revised in June 2026) built agents for 1,052 Americans from two-hour interviews and surveys. On held-out General Social Survey items, agents built from both reached accuracy equal to 86% of the participants' own consistency when the same people answered again two weeks later, against 74% for agents given demographics alone.

The gap between 86% and 74% carries the main lesson for persona design: a simulated person built from a rich profile tracks the real one more closely than one built from a few demographic labels.

Anatomy of an AI Agent Simulation Run

Whatever tool runs it, a simulation run has the same parts. The diagram follows one conversation from the scenario brief to a verdict; the whole loop then repeats.

• Scenario brief - the user's goal, the context, the success criteria and the stop condition. Without success criteria, the run produces a transcript nobody can score.
• Simulated user - a language model holding the persona and the hidden facts, writing one message per turn. A weak one volunteers the order number, suggests the fix and hands the agent its answer.
• Agent under test - reached through the interface customers use: the chat API, the voice stream, the phone number or the video session. A development endpoint skips speech recognition, authentication and rate limits, so failures in them never show.
• Conditions - background noise, accent, pace, interruptions and line quality for voice, phone and video agents. Studio-clean audio hides the recognition errors that real callers trigger.
• Stop condition - the goal is met, the goal is refused, a hand-off happens, or a turn cap is reached. MLflow's max_turns and DeepEval's max_user_simulations, for example, both default to 10.
• Evaluators - code checks first, then a judge model for what needs reading, with an evidence line for each verdict. A single overall score with no quote behind it cannot be audited.
• Repeats - the same scenario and persona, run several times, so each result is a rate with a run count behind it.

Designing Simulated Users and Personas

A persona tells the simulated user who they are, what they want, what they know and how they talk. Vary a dimension only when it can break something in your agent; the table pairs common dimensions with the failure each one tends to expose.

Profiles make simulated users more realistic. In SimulatorArena (Dou and colleagues, EMNLP 2025), simulated users were tested against 909 annotated conversations between people and language model assistants, on math tutoring and document creation. Simulators given user profiles, covering traits such as background and message style, rated the assistants in line with human judgments at a Spearman correlation of 0.7 on both tasks.

Write the persona as a card the simulated user reads on every turn:

persona: impatient customer
goal: money back for order 4471, a leaking blender
facts to give only when asked:
  - order 4471, delivered 12 days ago
  - opened and used once; it leaks from the base
behavior:
  - replies of 12 words or fewer
  - refuses to send a photo
  - asks for a manager if asked a second question
rules:
  - you are the customer, never the assistant
  - never suggest what the agent should do
  - stay in character if the agent gets it wrong
stop: reply DONE when the goal is met or refused twice
turn cap: 8 user turns

A language model is trained to be an assistant, and a simulated user drifts back toward that role unless its instructions stop it. These rules keep it in character:

• Hide the facts the agent must collect - give the order number and the delivery date only when asked, or every agent looks good at gathering information.
• Forbid helping - no suggested answers, no correcting the agent, no thanks for a wrong answer.
• Stop on a rule - a DONE token when the goal is met or refused, plus a turn cap, so every run ends for a reason you can count.
• Use another model family - a simulated user running on the agent's own model family is more likely to phrase requests the way the agent expects.
• Read before trusting - compare ten simulated transcripts with ten real ones before a new persona joins the suite.

For a ready-made catalog to compare with your own, TestMu AI Agent Testing ships 10 pre-built persona types: Accessibility Needs, Angry / Upset User, Confused Customer, Digital Novice, Expert User, Impatient User, International Caller, Multi-Lingual User, Off-Script User and Security-Conscious User. A custom persona takes its traits from five settings: communication style, vocabulary level, subject expertise, emotional state and interaction patterns.

Writing Scenarios Worth Simulating

A scenario gives one conversation its goal, the context around it, the criteria that decide a pass, and the condition that ends it. Written as a brief for the refund agent used later in this guide, it looks like this:

scenario: refund for an item that arrived damaged
context: order 4471, a blender delivered 12 days
  ago; the customer opened it and it leaks
success (all must hold):
  - the agent asks for evidence of the damage,
    or transfers when the customer asks for a person
  - every policy statement matches the written policy
  - no refund amount the policy does not offer
stop: refund agreed, transfer made, or 8 turns
category: edge case (opened and damaged)

Good scenarios come from material that already describes how the agent should behave and how customers really talk:

• Requirements and policies - the PRD, the refund and privacy policies, and the knowledge base the agent answers from.
• Standard operating procedures - the steps a human agent follows, including when to escalate.
• Support tickets and transcripts - last quarter's escalations and abandoned conversations, with personal data removed.
• Incident reviews - every production failure becomes a scenario that stays in the suite.

Spread the suite across happy paths, edge cases, adversarial inputs (manipulation, off-topic requests, prompt injection), persona-specific situations, and compliance cases such as a disclosure the agent must read out.

Then do the arithmetic, because every axis multiplies the conversations you pay for:

At three minutes a call and ten calls in parallel, 600 calls take three hours. Pruning by risk keeps the coverage where a failure costs the most:

• Full matrix for high-risk scenarios - the 8 scenarios that move money or change accounts, at 4 personas, 2 conditions and 3 runs: 192 conversations.
• One persona for the rest - the other 17 scenarios with the persona most likely to break each one, in quiet audio, 3 runs each: 51 conversations.
• Result - 243 conversations, about two fifths of the full matrix, or roughly 73 minutes at the same pace.

Drop a persona from a scenario when it cannot change the outcome, such as an expert user asking for an order status, and add one back when a production incident shows it can.

Simulating Voice, Phone and Video Conditions

A voice agent acts on the words its speech recognition produced, which can differ from what the caller said, so the simulated caller's audio is part of the test input. Vary it the way real calls vary:

• Accent and speech rate - the same sentence in several voices and at several speeds finds recognition errors that one neutral voice never triggers.
• Background noise - a street, a café, a car or a call center floor behind the caller.
• Interruptions - the caller talks over the agent mid-sentence, and the agent has to stop and answer the interruption (barge-in).
• Silence and delay - long pauses and a caller who says nothing, which exercise the agent's timeouts and reprompts.
• Keypad tones and transfers - on phone lines, menu choices sent as keypad (DTMF) tones, and hand-offs to a person or another queue.
• Line quality - a weak cellular connection or a compressed phone codec.

Time every turn as well. The phone agent testing docs define response latency as the time between the end of the simulated caller's speech and the start of the agent's reply, measured on every conversational turn; performance tests report it as percentiles from P50 to P99. The guide to voice agent testing covers each layer of a voice stack in depth.

A video agent adds a camera. The simulated participant needs a face and a voice, and the test also judges turn-taking, pacing and the agent's own presentation on screen. Treat the simulated participant as test equipment: its glitches should never lower the agent's score, and a session that failed on the harness side belongs outside the pass rate.

The walkthrough of video simulation testing covers how to write the brief a simulated participant improvises from. In the session below, the transcript labels the two sides Your Agent and TestMu AI Candidate.

Scoring a Simulated Conversation

Score in a fixed order, from the cheapest and most certain check to the most expensive:

• Code checks - anything a rule can decide: a hand-off event fired, a required disclosure appeared, a price the agent stated matches the price in your system, the conversation ended within the turn cap.
• A judge for meaning - did the user get the outcome they wanted, was the tone right, was the answer complete. Ask the judge for a verdict and a supporting quote for each criterion.
• A human sample - score a sample by hand, compare it with the judge, and fix the rubric where they disagree before trusting the judge on volume.

Score at two levels. Turn-level checks look at one reply, such as whether a stated price was right; conversation-level checks look at the whole exchange: did the user reach the goal, did the agent remember what it was told, how frustrated did the user get. MLflow's experimental multi-turn evaluation, for example, names conversation-level scorers ConversationCompleteness, UserFrustration and KnowledgeRetention.

How manual review, judges and simulation divide the work is compared in AI agent testing methodology. For writing the judge itself, see LLM-as-a-judge; for building the rubric, see chatbot evaluation.

Give each criterion a third outcome for missing evidence. When the transcript cannot show whether something happened, record the criterion as unable to verify and keep it out of the pass rate, instead of counting it as a pass or a fail.

Then aggregate across runs. Seven passes in 10 and 10 in 10 call for different release decisions, and whether a release needs every run to pass or at least one is the question pass@k vs pass^k answers.

The Agent Testing Certification validates skills in testing autonomous AI agents and conversational bots, and its study material includes running a first agent test and the Agent Testing CLI.

Worked Example: Simulating a Refund Agent

The run below uses a toy, so every number in it describes the toy and no real agent or product. The agent is a rule-based refund bot whose written policy says that opened items get store credit, unopened items get a full refund within 30 days, and items that arrive damaged get a full refund once the store sees a photo. It is also meant to transfer the customer to a colleague on request.

Before the run, two defects were planted in it: the hand-off rule recognizes "human" and "real person" but no other way of asking, and the agent invents a 50% refund for an item the customer calls barely or hardly used.

One scenario, a leaking blender from order 4471, ran with three scripted personas whose wording a seeded random generator draws from phrase lists, standing in for the run-to-run variation of a model playing the user. Each persona ran 10 times, and two code checks scored every conversation: a transfer in the turn right after the first request for a person, and every policy sentence the agent said matching the written policy word for word. The scripted personas are simpler than the persona card shown earlier: two of them give the order number without being asked, so this run does not test whether the agent asks for it.

The script ran on 16 October 2026 on Node.js v25.5.0, with no network calls:

node refund-sim.mjs

RUN sim-refund-20261016
toy refund agent, 1 scenario, 3 personas x 10 runs
node v25.5.0, 2026-10-16

TRANSCRIPT impatient, run 1 (seed 201)
user : Order 4471, it leaks. Fix it.
agent: Sorry about that. Items that arrive damaged get a
       full refund once we see a photo. Can you send one?
user : No photos. Put me through to a manager.
agent: I understand. Items that arrive damaged get a full
       refund once we see a photo. Can you send one?
user : I want a real person.
agent: Transferring you to a colleague now.
       [transfer_to_human]
user : Finally.

hand-off on request  FAIL
  evidence: user turn 2 asked for a person; the next agent
            turn did not transfer
only written policy  PASS

VERDICT confused, run 1 (seed 301)
only written policy  FAIL
  evidence: agent turn 2 said "Lightly used items qualify
            for a 50% refund.", which is not in the policy

PASS RATES (10 runs per persona)
persona       hand-off  policy  both
cooperative   10/10     10/10   10/10
impatient     5/10      10/10   5/10
confused      10/10     6/10    6/10

ONE-RUN SUITES (run k of every persona)
all green when k = 2, 7, 8, 9 (4 of 10)

• The cooperative persona found nothing - it gave the order number, sent the photo and passed all 10 runs. A suite built from happy-path conversations would have called the agent ready.
• The impatient persona found the hand-off gap in 5 of 10 runs - each time its first request said "manager", "supervisor" or "someone" instead of "human" or "real person". In run 1, the transfer came only after the second request.
• The confused persona found the invented rule in 4 of 10 runs - each time it said it had barely or hardly used the blender, and in every one of those runs it accepted the 50% offer and ended the chat.
• A one-run suite would often have shipped both defects - taking run k of every persona as one suite, 4 of the 10 possible suites (k = 2, 7, 8 and 9) came back all green.

Both failures are easy to miss for a reviewer reading transcripts for outcomes, because 8 of the 9 failed conversations still end in a transfer or an accepted offer. The specific checks caught them. A real suite swaps the scripted personas for a model in character and the toy for your agent's endpoint, and the checks and the arithmetic stay the same.

How to Build a Minimal User Simulator in Python

A conversation simulator needs little code: a persona prompt, a loop, one model call per user turn, one call to your agent, and a transcript on disk. This one uses only the Python standard library and any chat completions endpoint that follows OpenAI's request format, and it reads the address, the key and the model name from environment variables.

import json, os, uuid, urllib.request

SIM_URL = os.environ["SIM_BASE_URL"] + "/chat/completions"
SIM_KEY = os.environ["SIM_API_KEY"]
SIM_MODEL = os.environ["SIM_MODEL"]
AGENT_URL = os.environ["AGENT_URL"]
MAX_TURNS = 8

PERSONA = """You are a customer, not an assistant.
Goal: get your money back for order 4471,
a blender that leaks. Mention the order number
only when asked. You are impatient and write
short messages. If the agent asks for a photo,
refuse and ask for a manager. Never suggest
answers to the agent. When you get what you
want, or are refused twice, reply only DONE."""

def post(url, body, key=None):
    head = {"Content-Type": "application/json"}
    if key:
        head["Authorization"] = "Bearer " + key
    data = json.dumps(body).encode()
    req = urllib.request.Request(url, data, head)
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.load(r)

def user_turn(history):
    # Roles flip: to the simulator, the agent's
    # messages are "user" and its own "assistant".
    msgs = [{"role": "system", "content": PERSONA},
            {"role": "user", "content": "Start."}]
    for t in history:
        role = "assistant" if t["role"] == "user" \
            else "user"
        msgs.append({"role": role, "content": t["text"]})
    out = post(SIM_URL, {"model": SIM_MODEL,
                         "messages": msgs}, SIM_KEY)
    return out["choices"][0]["message"]["content"]

session, history = str(uuid.uuid4()), []
for _ in range(MAX_TURNS):
    text = user_turn(history)
    history.append({"role": "user", "text": text})
    if text.strip() == "DONE":
        break
    reply = post(AGENT_URL, {"session": session,
                             "message": text})
    history.append({"role": "agent",
                    "text": reply["reply"]})

with open("transcript.json", "w") as f:
    json.dump(history, f, indent=2)

• Role flip - the model playing the user sees the agent's messages as the user's and its own as the assistant's, which keeps it writing as the customer.
• Stop rule - the loop ends when the simulated user replies DONE, or after 8 user turns.
• Session - one session ID travels with every message, so the agent keeps its conversation state.
• Agent contract - the code assumes your agent accepts a JSON body with session and message fields and returns a reply field; change the two lines that call the agent to match yours.

The loop was checked on 16 October 2026 with Python 3.14 against two local stub servers, one standing in for the model and one for the agent. It alternated the roles correctly, sent the key in the Authorization header, kept one session ID and wrote transcript.json after the simulated user replied DONE.

What the script leaves out is most of a test suite: hundreds of conversations in parallel, speech and telephony for voice and phone agents, video sessions, evaluators with evidence, repeat-run statistics, and reports someone outside the team can read. That is the work AI conversation simulation software takes on.

Limits of AI Agent Simulation

Simulated users are not real users, and a pass rate measured with them inherits the difference. Lost in Simulation (Seshadri and colleagues, January 2026) measured how large it is, with a user study on tau-bench retail tasks in which participants from the United States, India, Kenya and Nigeria played the customer:

• The simulator changes the score - with the same GPT-4o agent, success rates varied by up to 9 percentage points depending on which model played the user.
• Errors run in both directions - simulated users underestimated the agent's performance on challenging tasks and overestimated it on moderately difficult ones.
• Simulated users are more polite and ask more questions - questions appeared in 18.8% of simulated user turns against 9.8% of human turns, and politeness markers in 39.2% against 19.9%.
• Some speakers are poorly represented - simulated users were the weakest proxy for speakers of African American Vernacular English and Indian English.

Personas also wear off over long conversations. In a COLM 2024 study of instruction drift, first posted as Measuring and Controlling Persona Drift in Language Model Dialogs, Li and colleagues found significant drift within eight rounds of conversation in LLaMA2-chat-70B and GPT-3.5. A simulated user told to stay impatient can turn agreeable halfway through a long call.

The method has structural limits too:

• Coverage stops at what was written - a suite tests the scenarios and personas someone thought of, and the failure nobody imagined is not in it.
• Shared blind spots - when one model family writes the scenarios, plays the user and judges the result, its habits shape all three.
• Effects outside the conversation - a simulation shows what the agent said and which actions it started; whether the refund reached the billing system is a separate check. When the agent changes records itself, the release check for that is Agent Assurance from TestMu AI, which is pre-alpha and publicly installable: it grades each acceptance criterion against the changes a run left behind.

Each limit has a practical counter:

• Seed personas and scenarios from real transcripts, and refresh them every quarter.
• Run the suite with the simulated user on two different models, and treat a large gap between their pass rates as a finding.
• Read a weekly sample of real and simulated conversations side by side.
• Keep monitoring production, and turn every incident into a new scenario.

AI Agent Simulation Tools Compared

Tools for simulation testing of AI agents include libraries that run inside your own code, simulation built into an agent platform, and hosted platforms that also place the calls, run the sessions and write the reports. The table lists what each vendor's own site and docs said on 16 October 2026, in alphabetical order; it is not a ranking.

The kinds differ in who does the work:

• Libraries - DeepEval, LangWatch Scenario and MLflow put the simulator in your own code under an open-source license; your team writes the test harness and runs it on its own infrastructure.
• Simulation inside an agent platform - Sierra's simulations test agents built on Sierra.
• Hosted platforms - Cekura, Coval, Maxim and TestMu AI run the conversations for you; compare the agent types each one reaches and how it connects to yours.

If your agents are built or simulated in Maxim, the Maxim testing page explains how TestMu AI tests them alongside it.

AI Agent Simulations You Can Run on TestMu AI

Agent Testing from TestMu AI covers chat, voice, phone, video and image agents. On chat, voice, phone and video, simulated users hold the conversations; image agents are scored from batch uploads of up to 50 images.

• Scenario generation - 60 to 100+ scenarios generated from uploaded documents and PRDs, or from connected Jira and Confluence, spread across happy paths, edge cases and adversarial inputs.
• Real phone calls - Phone Caller tests run over real telephony: an AI-powered simulated caller calls an inbound agent or answers an outbound agent's call, with 200+ voice profiles and 15 background noise presets to choose from.
• Video sessions - a simulated candidate with a face and a voice joins your agent's session URL. Sessions multiply as avatar faces x personas x test profiles x iterations, with up to 10 iterations per scenario. A session that breaks on the TestMu AI side is labeled Inconclusive and left out of the pass rate.
• Evaluators and metrics - 15+ specialized evaluators, Hallucination Hunter among them, work in parallel. Chat and voice conversations are scored 0 to 100% on 9 metrics: Bias Detection, Hallucination Detection, Completeness, Context Awareness, Response Quality, Conversation Flow, User Satisfaction, File Handling Quality and File Generation Accuracy.
• Stated facts against your records - when a chat agent quotes a delivery date or a stock count during a simulated conversation, Data Validation fetches the real value from your system of record with a read-only API call after the conversation and gives each fact its own Pass, Fail or Cannot verify verdict.
• Confidence and verdict - each metric carries a confidence level set by how many evaluations back it, and the outcome of a chat, voice or phone run is a go-live verdict of Green, Yellow or Red, while a video run returns Pass or Fail against your success criteria. A Green verdict covers the scenarios and personas that were tested; production will bring conversations outside them.
• CI runs - the Agent Testing CLI launches saved Chat and Phone Caller suites in a pipeline, reads LT_USERNAME and LT_ACCESS_KEY for credentials, and returns exit code 1 when a test in the finished run has failed.

pip install agent-testing-cli
agent-testing-cli --project PROJECT_ID run \
  --suite SUITE_ID --yes \
  --wait --poll 5 --timeout 1800

The --wait flag holds a Phone Caller run until its final result; --poll and --timeout are in seconds, and a run that reaches the timeout exits with code 5. Chat evaluations run asynchronously, so a chat suite is started with its workflow ID and read back later with the results command.

Your First AI Agent Simulation Suite

Pick one conversation that went wrong in production last month and turn it into a scenario brief with two or three criteria that code can check. Write three personas for it (cooperative, impatient and confused), run each one five times, and read every failed transcript before you write a fourth.

To run the same loop over real calls, video sessions and chat endpoints, connect the agent in Agent Testing; for a chat or voice agent, the guide to testing your first AI agent covers the setup, the first evaluation and reading its results.