How to Generate Test Cases for AI Agents That Act

What Is AI Agent Test Case Generation?

AI agent test case generation is the automated production of test scenarios for an AI agent under test. Each scenario holds the input sent to the agent, the starting state it assumes, and acceptance criteria a grader checks after the run. Generators work from the agent's requirements, its code and tool definitions, or recorded traffic.

The phrase also gets used for the opposite job, so it helps to know which one you're looking at:

• The agent as the generator - an AI agent reads requirements and writes test cases for your web or mobile app. The app is the system under test. The roundup of AI test case generation tools covers that job, and the free AI test case generator drafts cases from a plain-language description.
• The agent as the system under test - the agent itself is being tested, and the test cases check its tool calls, its side effects, and its replies. That's this article.

Generation exists because writing these cases by hand doesn't keep up. The authors of Agent Seer (arXiv, June 2026) put it directly: building tool-use scenarios by hand "demands deep domain expertise, does not scale across tool ecosystems, and produces static benchmarks that cannot track evolving APIs." Their pipeline drafts scenarios from a single MCP specification, with no examples and no live tool access.

A generated suite still needs a capability spec to check against. The guide to agent functional testing covers writing cases by hand from that spec. This one covers producing them automatically and checking that they can be graded.

Anatomy of a Gradable Agent Test Case

A generator fills the same fields every time. The question is whether what it puts in each field gives a grader something to check after the run.

The last row matters most. A criterion with no verification method gets graded by whatever is left over, which is usually a model reading the agent's reply, and that's the weakest evidence you have.

The Agent Assurance scenario docs make these fields explicit. Acceptance criteria are graded independently, a forbidden list holds leakage and hallucination tripwires, and verification_requires names the evidence each check depends on.

Generation Inputs Compared: Specs, Code, Tool Schemas, and Traces

What you generate from sets the limit on what the suite can find. Each source tells the generator something different and leaves something out.

MANTRA (arXiv, May 2026) generates the check alongside the task. From natural-language manuals and tool schemas, it builds a symbolic world model and a set of trace-level compliance checks, then uses SMT solving to confirm the two are consistent. Its authors built a suite of 285 tasks across 6 domains, scaling to manuals over 50 pages long.

Schemas also hint at where generation struggles. In the Agent Seer study, parameter schema complexity was the strongest correlate of variation in scenario quality, and the number of tools in the suite played a smaller role.

The Agent Assurance overview docs state the limit of a document-only suite plainly: "Documentation is specification evidence, not proof of implementation." Pair at least one source of intent (a PRD or policy manual) with one source of implementation (code, schemas, or traces).

The Generation Workflow, Step by Step

The order below front-loads the work generators can't do for you, so the generated output arrives with something to check against.

• List the tools and their side effects - every tool the agent can call, whether it writes, and what it writes to. If you can't tell whether a tool writes, record it as unknown rather than guessing from its name, which is how the Agent Assurance discovery docs treat it too.
• Pull out every rule with a number in it - approval limits, retry caps, rate limits, required fields. Each number gives you at least two cases: exactly on the limit and one unit past it.
• Generate each class separately - functional (happy path, negative, boundary, integration, state), non-functional (latency, cost, repeatability), and adversarial (prompt injection, data exfiltration, policy violation). One mixed batch can let happy paths crowd out the rest.
• Ground each case in fixture data - replace placeholder IDs with records that exist in staging and declare them as preconditions.
• Attach a check to every criterion - a read-only query, a tool-call log, a file on disk, or a judge on the reply.
• Deduplicate - merge cases that test the same rule with reworded input.
• Run a small slice first - read which criteria couldn't be checked before you pay for the full suite.

Step 5 changes the output most. A typical generated case:

- name: Refund a damaged order
  input: "Refund my order, it arrived broken."
  expected_output: "The agent confirms the refund has been processed."

Below, the same request becomes a boundary case that can be graded, written with the field names from the Agent Assurance scenario format. Its precondition is a staging order, A-1184, that totals one unit above the approval limit:

id: SC-021
feature_id: refund-request
class: functional
category: boundary
title: Refund one unit above the approval limit is escalated, not issued
goal: >-
  Refund order A-1184 in full. It arrived broken.
input:
  kind: text
  attachments: []
expectation:
  acceptance_criteria:
    - id: AC-1
      statement: The agent does not issue a refund above the approval limit.
      check: mcp_probe
    - id: AC-2
      statement: The reply says the refund needs approval and offers a handoff.
      check: llm_judge
  forbidden:
    - claims the refund was issued
  output_kind: text
  mcp:
    - server: billing
      tool: issue_refund
      expect: not_called
verification_requires:
  - type: mcp
    server: billing
    op: issue_refund
repeat: 1
timeout_seconds: 120
tags: [refund, boundary]

AC-1 is settled by a probe against billing. AC-2 is a reply check, and the forbidden line catches the agent claiming a refund that billing never recorded.

Note: TestMu AI Agent Assurance writes these scenarios for you from an agent's code or PRD, across functional, non-functional, and adversarial classes, then grades each criterion against what the run changed. Try TestMu AI free!

Edge Cases Generators Miss

The authors of EdgeGen (arXiv, September 2026) found that existing synthetic task generators "often produce generic tasks that ignore an agent's underlying state or database."

EdgeGen extracts compliance rules from the agent's spec and generates database-grounded tasks designed to violate them. On the tau2-bench airline domain, finetuning on that data improved mean progress by 2% to 42%, and harness optimization improved it by 10% over a human-curated harness for the Gemma-4-e4b model.

Generate these on purpose, one class at a time:

• Rule violations - for every must-never rule, a request that tries to break it using a real record.
• Config boundaries - the limit, one unit past it, and one unit under it. Off-by-one comparisons show up here.
• Tool failures - a timeout, an empty result, an error response, a partial write. The criterion is that the agent reports the failure instead of claiming success.
• Stale or conflicting state - the order was already refunded, or the record changed between the agent's read and its write.
• Instructions inside data - a ticket body, order note, or tool result that tells the agent to do something else. The AI agent red teaming test plan covers tool misuse in depth.
• Mid-conversation changes - the user changes the amount or the order number on the third turn.

Reviewing a Generated Suite Before You Run It

A generated suite is a draft. Read the criteria, not just the titles, and fix these before you spend a run:

• Criteria that restate the goal - "the agent refunds the order" as both input and expected result checks nothing.
• Reply-only checks on write actions - if the case expects a write, at least one criterion has to read the system it wrote to.
• Pinned tool order - an exact call sequence fails correct runs that take a different route. Assert the end state and which calls must or must never happen.
• Wrong argument values - the Agent Seer authors found argument value accuracy was the dominant failure among imperfect scenarios, "a sub-dimension invisible to coarse-grained name-match metrics." Check every ID, amount, and date against your fixtures.
• Missing fixtures - a case that names an order staging doesn't have fails for the wrong reason.
• Near-duplicates - ten rewordings of one happy path raise the scenario count without adding coverage.
• Write tools pointed at production - adversarial cases try destructive calls by design, so run the suite against staging.

Then do a small run and count the criteria that came back unverifiable. That count depends heavily on what your invocation exposes. The Agent Assurance docs note that a profile whose hook returns only an answer string "leaves most criteria Unable to Verify." Expose tool calls and state, and that count drops.

In this Agent Assurance run of a release-notes agent, the Unverifiable expectations panel lists 4 expectations across SC-004 and SC-007 that couldn't be checked because the run had no view of the agent's tool calls. The coverage-gaps panel sits beside it, broken down by category. The broader question, deciding what counts as a correct result when there's no fixed expected output, is the test oracle problem.

AI Agent Test Case Generation With Agent Assurance

TestMu AI Agent Assurance tests how your agents actually behave across workflows, tools, and actions, and catches failures and vulnerabilities before they ship. It runs from your terminal as rook and covers most of the workflow above:

• Discovery from code or a PRD - exploration reads source code, prompts, manifests, tool declarations, MCP server declarations, and PRDs. Discovery doesn't invent missing facts.
• 3 classes, 18 categories - functional (happy path, negative, boundary, integration, state handling), non-functional (performance, token economy, reliability, quality), and adversarial, which covers prompt injection, jailbreak, data exfiltration, PII leakage, and five more. Scope a run with --class, --category, and --total.
• Reviewable YAML - every scenario is a plain file you can read in a pull request, exclude, or delete. Completed runs keep a snapshot of the definitions they executed.
• Three verdicts - each criterion is graded Pass, Fail, or Unable to Verify against recorded tool calls, changed files, artifacts, and read-only checks. The rook README states that unverifiable results are excluded from the pass rate's denominator rather than counted against you.

The summary credits discovery to src/agents/release.mjs with high confidence and lists 2 features and 28 scenarios for the agent. The HTTP invoke profile on the right is the one input Agent Assurance can't derive.

A first run in the terminal UI, after rook login:

/project
/explore .
/agent
/profile add staging
/generate --class functional,adversarial --total 24
/scenarios list
/sync
/run
/ui

Agent Assurance is pre-alpha, so commands and stored file formats can still change. The agent's writes are real and can't be rolled back, so point it at staging.

In CI, a finished run exits 0 whether its scenarios passed or failed. Gate the build on the verdicts in rook report --json.

Next Steps

Start AI agent test case generation with the agent's riskiest write action. Generate cases for it from both the spec and the code, and give every criterion a check that reads the system before you run anything.

To have the suite generated from your agent's code instead, follow the Agent Assurance quickstart.