End to End Agent Testing: How to Test the Whole Path

Most teams shipping an agent already have an end to end suite. It walks a browser through a checkout, asserts the confirmation page appeared, and passes. Then they point the same instinct at the agent and find there is nothing to click, no fixed route to record, and no page whose contents settle the question.

The practice still applies. What changes is where the assertions live and what counts as the end of the path.

End to end agent testing exercises the agent across its complete path: the request a user makes in plain language, the plan the model forms, every tool it calls, the change those calls make in a real system, and the response that comes back.

The phrase is borrowed from application testing, and the borrowing is only partly accurate. The table below maps what each term means in each world.

If your agent operates a web application, the classic practice still covers that application. For a refresher on that layer, the end-to-end testing guide covers the fundamentals, and the Cypress end to end testing tutorial walks through a worked example.

Three habits that make a browser suite reliable make an agent suite useless.

• Scripting the steps - a browser test records the route because the route is stable. Pin an agent's tool sequence and the test fails the first time the model picks an equally correct alternative.
• Asserting on the surface - the DOM tells you what rendered. It cannot tell you which tools ran, what arguments they carried, or whether a stated limit was respected.
• Trusting the confirmation - a confirmation page is produced by the system. An agent's closing message is produced by the same model that may have failed, which makes it the weakest evidence in the run.

That third one is the expensive habit. An agent can state that it issued a refund and notified the customer, in fluent and well-formed prose, when only the refund happened.

Separating the run into stages is what makes a failure attributable rather than just visible.

1. Request       "Refund order A-1183, the item arrived damaged."
2. Plan          the agent decides which tools to use, and in what order
3. Tool calls    lookup_order, then issue_refund, then notify_customer
4. Effect        a refund row exists, an email is queued, an audit entry is written
5. Response      "I've refunded the order and let the customer know."

The test asks one question at each boundary, and one across them:
does stage 5 accurately describe stage 4?

Stages one and two are cheap to get wrong and cheap to fix. Stages three and four are where money moves. Stage five is where a failure becomes invisible, because it is the only stage most teams read.

Note: An agent that acts on real systems needs checking against the systems, not against its own summary. TestMu AI runs that layer. Try it free.

Each stage has an assertion that holds regardless of the route the agent took, and a tempting assertion that will make the suite flaky.

The last row carries most of the value and is the one teams skip, because checking it means reading the effect and the response together rather than either alone.

• Pick the journeys where a failure would be expensive, and stop there. Paths that move money, change customer records, or send something outward earn an end to end test; the rest do not.
• Write each scenario as a goal plus criteria, not as steps. The goal is what you would tell a colleague; the criteria are what you would check afterwards.
• Give the run a way to observe the effect, whether that is database access, an API to read back from, or an audit log the agent writes.
• Add the negative cases early, such as a request just past an approval limit, a missing record, or a tool that returns an error.
• Add adversarial cases too, since instructions hidden in the content an agent reads are a first-class risk rather than an edge case.
• Keep the lower levels doing the narrow work, so the end to end suite stays small enough to run often.

Where a system hands work between several agents, the same discipline extends across the handoffs, which the guide to multi agent testing covers in detail.

Two constraints shape how an agent suite fits a pipeline, and neither applies to a browser suite.

The runs are not free and not reversible. Every execution spends tokens and performs real actions, so point the suite at staging with test data you can regenerate, and know which write-capable tools the agent can reach before the first run.

Two failure kinds need separating. A run where the agent did the wrong thing is a defect in your system. A run that could not reach or invoke the agent is infrastructure. Reporting them identically sends engineers looking for a bug that is not there.

Because the suite is slow, most teams run it on merge to the main branch and on a schedule rather than on every commit, keeping faster checks on the pull request itself. The approach in getting E2E coverage on every PR is a useful comparison for the layer underneath.

TestMu AI covers this through Agent Assurance, which splits by how an agent fails. Agents that act on systems are graded on the truth of the effect; agents that talk to people are graded on the quality of the reply, on the AI agent testing platform.

For the acting kind, three decisions line up with the stages above. Scenarios are derived from your codebase rather than authored by hand, so the suite reflects what the agent can actually do. Verdicts are recorded per criterion rather than per scenario, so a run that half worked reports as such. Anything the run could not check is reported as unverifiable and left out of the pass rate, which keeps a green number from mixing what was proved with what was assumed. This category is pre-alpha at the time of writing.

Where the agent drives a browser, the application layer still needs its own verdict, and Kane CLI checks the rendered result in a real Chrome browser from a natural-language objective.

Take the single agent journey that would cost you most if it silently half-worked, and write down what must be true in your systems once it finishes. If you can check those things without reading the agent's closing message, you have your first end to end agent test.

Add the negative and adversarial cases next, keep the suite small, and put it on merge rather than on every commit. For how this sits inside a wider practice, see the guide to agentic quality assurance, and for the production half of the picture, agent observability. For the narrower layer beneath this one, agent functional testing covers deriving test cases from a capability spec.

To try this against your own agents, create a free TestMu AI account and start with the getting started documentation.