What is agent assurance?

Agent assurance is the practice of proving an AI agent is safe to ship, rather than assuming it from the agent's own summary of its work. For an autonomous agent, that means invoking it for real, watching what changed while it ran, and grading each individual claim against that evidence. Agent Assurance is TestMu AI's product for doing this.

How is this different from an eval framework?

An eval framework asks you to write the scenarios and then scores the agent's final message with a language model judge. Agent Assurance derives the scenarios from your codebase and grades the effect: files that changed on disk, artifacts produced, and tool calls checked against the agent's own declared tool surface. Our primer on AI agent testing covers the wider landscape.

What is the assurance gap?

The assurance gap is the percentage of criteria a run could not verify. It is reported beside the pass rate so you know how much of that pass rate was actually observed rather than inferred. Most teams have never computed this number for their own agents.

Why report what you could not check? Does that not look bad?

A pass rate that mixes proved and assumed results cannot be trusted to move. Publishing the blind spot is what makes the rest of the number worth reading. It is also actionable: the gap is mostly a property of your agent, so an agent that records what it does is far more verifiable than one that does not.

Do I have to write the tests?

No. Agent Assurance reads the codebase and derives the suite. The only thing you supply is how to invoke the agent, as a command, an HTTP endpoint, or an MCP server. It parses what you paste, shows you what it understood field by field, and asks you to confirm before anything is spent.

Does it work with non-deterministic agents?

Yes. It does not pin the execution path, which is what makes conventional tests flaky against agents. It grades criteria against evidence instead, and reports flakiness as its own category, separate from regression, because the two call for opposite responses.

Will it change my data?

It invokes your agent the way a user would, so the agent's writes are real and cannot be rolled back. Before a run it tells you how many write tools the agent declares and asks once, and grants are per target. Judges are instructed to verify without changing anything. Point it at staging.

What kinds of agents does it cover?

Agents that act: coding agents, workflow agents, back-office agents, and anything else that calls tools, writes files, or hits APIs. If your agent talks to a human instead, through chat, voice, phone, or image, that is the job of conversational agent testing, which grades the reply rather than the effect. It is framework agnostic, so it works the same way across the common LLM agent frameworks.

Can I gate my build on it?

Yes. The headless subcommands return meaningful exit codes: fail the build on exit 2, which means the agent did something wrong. Exit 1 means the harness could not test the agent at all, which is infrastructure rather than a finding about your agent. Nothing unverifiable fails the build.

Do I need my own model API keys?

No. Models run in the TestMu AI controller, so no provider keys sit on your laptop or your CI runner. Run state stays in your project as plain files. It needs Node 20 or newer, a TestMu AI account for credits, and whatever your agent itself needs to run.

How does this relate to Kane CLI and KaneAI?

Same evidence philosophy, different subject. Kane CLI is a browser agent that proves your application's UI works, and KaneAI authors tests for applications in natural language. Agent Assurance proves the agent itself works.

The Autonomous Agent category is not generally available yet, so access is by waitlist. Join it from any section on this page and we will get in touch. In the meantime, agent testing for chat, voice, and phone agents is available today, and pricing covers the wider platform.

Prove Your AI Agent Did What It Said

Point Agent Assurance at an agent you own. It derives the suite from your code, invokes the agent for real, and reports what it could not verify.

See how it works

Automate Browser Flows from your
Terminal with Kane CLI

Explore Kane CLI

Trusted by 3M+ users globally at

+Read case study

Two Premises. Everything Follows.

Agent Assurance grades what your agent actually did, and publishes the share of it nobody could see.

Scenario verdict showing the agent's claim beside the tool calls and file changes actually observed

Grade the effect, not the account

An agent's account of what it did is the weakest evidence available about what it did. It is the one party with a reason to be wrong.

Run report showing pass rate beside the assurance gap and the split of pass, fail and unable to verify

Publish the blind spot

Anything Agent Assurance could not verify is reported as unverifiable. Never a quiet pass, never a guessed fail, never folded into the pass rate.

No tests to write

It reads the codebase and derives functional, non-functional, and adversarial scenarios for the agent it finds there.

Adversarial by default

Prompt injection, instruction override, and tool misuse are a first-class scenario family, not an add-on you configure.

Runs in your CI

Headless subcommands and exit codes that separate a broken agent from a harness that never got a look at it.

How Agent Assurance Proves an Agent Works

Four phases, from a folder you own to a verdict you can take to a release meeting.

DISCOVER

Your Codebase Is the Test Plan

Manifests, prompts, tool tables, and MCP servers. Where an agent declares nothing, it says so.

Deterministic discovery reads manifests and tool tables
Connects to MCP servers and asks what tools exist
Undeclared fields come back unknown, never guessed

GENERATE

A Suite You Did Not Have to Write

Functional, non-functional, and adversarial scenarios, each carrying its own gradable criteria.

Scenarios derived from code, not authored by you
Adversarial family covers injection and tool misuse
The criterion is the unit of judgement, not the test

RUN AND JUDGE

Graded on Effect, Not on Claims

The agent is invoked for real, and each criterion is judged against what actually changed.

Watches files, artifacts, and every tool call made
Checks calls against the agent's declared tool surface
Judges verify read-only and change nothing

REPORT

Two Numbers, Not One

Per-criterion verdicts with quoted evidence, plus the share of criteria nobody could check.

Pass, fail, and unable to verify are three verdicts
Unverifiable is excluded from the pass rate
Newly failing, newly fixed, and flaky are separated

Exit Codes

Why 1 and 2 Are Different Numbers

A build should stop for either, but only one of them is a finding about your agent. Exit 1 means the harness never got a look. This is the exit-code expression of the same honesty rule, and it is why nothing unverifiable ever fails your build.

0It ran, and nothing was wrongShip it

1It could not test the agentInfrastructure, not a finding

2The agent did something wrongFail the build on this one

3Not signed inAuth, not a finding

4Out of budgetRaise the cap and re-run

Fail the build on 2. Treat 1 as infrastructure and surface it loudly rather than swallowing it. Measure the assurance gap for a few weeks before you gate on it at all, then set a threshold you have earned.

What Each Approach Actually Proves

Most tools grade the transcript. Agent Assurance grades the effect, and says what it could not see.

Agent Assurance

Eval frameworks

LLM observability

Who writes the scenarios

Derived from your codebase

You do

Not applicable

What is graded

Each criterion, against evidence

The final response

Spans after the fact

Evidence used

Files, artifacts, tool calls

Transcript text

Traces and logs

Tool calls checked against the declared surface

Yes

Recorded, not judged

Verdicts available

Pass, fail, unable to verify

Pass and fail

No verdict

What could not be proved

Reported as a number

Silently folded into the score

Not measured

Adversarial coverage

Generated as a family

Only what you author

None

Runs before release

Yes, in CI

Yes

No, production only

Built for Every Layer of Agent Assurance

Engineers building agents

A generated suite in minutes without writing tests, plus a run-over-run diff that separates a real regression from a flaky scenario.

QA and test engineering

A unit of coverage that survives scrutiny. Criteria proved against evidence, and an explicit account of everything that was not checked.

Engineering leaders

Two numbers instead of one. The pass rate, and the assurance gap that tells you how much of that pass rate anybody actually observed.

Platform and DevEx teams

One assurance step that drops into CI for every agent your product teams ship, replacing a homemade eval script per repository.

Security and risk

Prompt injection and tool misuse become a repeatable test, with an exit code that fires when the agent is compromised.

Compliance and audit

Per-criterion evidence retained as plain files, alongside an explicit record of what could not be verified on each run.

Success Stories of TestMu AI (Formerly LambdaTest)

50%

reduction in test execution time

“HyperExecute is a highly reliable test execution platform and has excellent customer support.”

Sagar Uday Kumar

Sr. Engineering Manager

Three Surfaces. One Verdict.

Work in the terminal, gate the build in CI, and hand an auditor a sealed evidence pack.

The terminal

An interactive session for the engineer who owns the agent. Anything that spends shows its plan and asks first, so a suite never surprises you with its cost.

Your CI pipeline

Headless subcommands with exit codes that distinguish a broken agent from a harness that could not test it. Unverifiable results are reported, never used to fail a build.

A sealed evidence pack

Requests, responses, per-criterion verdicts and artifacts for a run, plus the record of what was not verified. The artifact a security review actually asks for.

Some Love from our Customers

As Best Egg expanded its product offerings and entered new markets, we knew our old testing infrastructure couldn’t keep up.
With support from Tenny Agustin, our Engineering Operations Lead, we modernized our approach with

TestMu AI

Best Egg

best-egg

Excited to Share My Learning Journey with Kane AI & Lambda Tool!
I'm pleased to announce that I've recently gained hands-on experience exploring Kane AI through the Lambda Tool and it’s been a fantastic journey of upskilling!

KaneAI

Suryateja Goud

suryateja-goud

See how is #Futureready to enable blazing-fast test orchestration seamlessly integrated with organizations' existing CI/CD platforms, using #Microsoft Azure.

TestMu AI

Microsoft India

MicrosoftIndia

View all reviews

More Reasons to LoveTestMu AI (formerly LambdaTest)

See how TestMu AI speeds up your testing with AI-native authoring, faster execution, and deeper test insights across web, mobile, and AI applications.

TestMu AI Named a Challenger in the 2025 Gartner® Magic Quadrant™

Read Report

TestMu AI recognized in The Forrester Wave™: Autonomous Testing Platforms, Q4 2025

Read Report

Wall of Fame

TestMu AI is #1 choice for SMBs and Enterprises across the globe.

Enterprise-Grade Security

We safeguard your data and AI systems with global security, privacy, responsible AI, and ESG standards.

Integrations

Works where you work, 120+ integrations with the tools your team relies on.

Users

3M+

Tests

1.5B+

Enterprises

18K+

Countries

132

As Seen On

Frequently asked questions

TestMu AI (Formerly LambdaTest)/Agent Assurance

TestMu AI forEnterprise

Get access to solutions built on Enterprise
grade security, privacy, & compliance

Advanced access controls
Advanced data retention rules
Advanced Local Testing
Premium Support options
Early access to beta features
Private Slack Channel
Unlimited Manual Accessibility DevTools Tests