AI Agent Assurance for Your Agent Deployments
Agent Assurance Tests How Your Agent Actually Behaves across Workflows, Tools, and Actions to Catch Failures and Vulnerabilities.
npm install -g @testmuai/rook@latest
AI Reliability Is a Repeatability Problem: Measure pass^k, Not Your Best Run
AI reliability is whether an agent succeeds every time, not once. Use the Reliability Envelope and pass^k to test AI agents and gate releases on evidence.
Published on:
On 18th November, Arena launched its Agent Arena Index. It's an index that takes the average of task completion, praise to complaints, steerability and bash reliability across over 2.9m user sessions.
It's a very useful signal. It's also an average, and averages are precisely where agent reliability problems are.
The research is now quite blunt about it. Princeton's "Towards a Science of AI Agent Reliability" (ICML 2026) has tested 15 models on two benchmarks and found that capability gains have only resulted in "small improvements in reliability".
The findings page goes on to state that over the 24 months of model development, accuracy has continued to go up whilst reliability has shown only small improvements.
So here's my thesis for QA leads and SDETs. Stop asking "how often does the agent get it right?" Start asking "how often does it get it right every time under the conditions it will meet?"
The first question gives you a demo. The second gives you a release.
TL;DR
AI reliability is the probability that an AI agent will complete the same task correctly every time it is asked. It is not the agent's best score. You measure it via pass^k across repeated runs, rephrased inputs and injected tool faults. You check the outcomes against the real system state and gate each release on that evidence. Accuracy is a measure of capability. Reliability is a measure of the worst case.
How Do You Measure AI Reliability for an AI Agent?
- pass^k: The chance that all k independent runs of the same task succeed. It falls as k grows, while pass@k rises, which is why pass@k vs pass^k decides whether a score describes a demo or a release.
- LLM evaluation metrics: Single-run scores such as pass@1 and a mean eval score measure capability. Reliability needs repeat-run metrics, a worst-case pass rate and failure severity, the measures covered in AI agent evaluation metrics.
- AI evals: AI evals count as reliability evidence only when each run is checked against systems of record and any judgment call goes to a judge from a different model family than the agent.
- Agent evaluation: LLM testing checks what a model writes. AI agent evaluation grades the whole agent, including its tools, data and permissions, so a reliability suite adds rephrased inputs and injected tool faults to repeated runs.
- Agent harness: An agent harness runs the agent against fixed scenarios, fixtures and pass criteria, which keeps the measurement deterministic even though the agent's own outputs vary from run to run.
- AI observability: AI observability records how an agent behaved after release. The Reliability Envelope tests how it will behave before release, and production incidents feed the envelope's rephrase and fault libraries.
- AI assurance: AI assurance turns repeated agent runs into release evidence. TestMu AI's Agent Assurance reports each criterion as Pass, Fail or Unable to Verify, and keeps unverifiable results out of the pass rate.
What Is AI Reliability and How Is It Any Different From Accuracy?
AI reliability is how consistently an AI system produces a correct outcome for the same task (across multiple runs and with realistic variation in inputs and tools). Accuracy is about what the agent can do. Reliability is about what it will do.
NIST uses the classic engineering definition in the AI Risk Management Framework. Reliability is the ability to perform as required without failure over a given time interval under given conditions. NIST also lists "valid and reliable" as one of the characteristics of trustworthy AI.
Note the inclusion of 'without failure' and 'given conditions'. Neither of those describes an average over a benchmark.
| Aspect | Accuracy | Reliability |
|---|---|---|
| Question it answers | Can the agent solve this task? | Will it solve this task every time? |
| Typical metric | pass@1, pass@k, mean eval score | pass^k, worst-case pass rate, failure severity |
| How it is measured | One run per task, averaged | Many runs per task, with variation; all must pass |
| What it hides | Run-to-run variance, brittle phrasing, fault handling | Little, if your conditions match production |
| Who needs it | Model selection, research | Release decisions, QA sign-off, risk owners |
Why Does pass@1 Overstate the Reliability of AI Agents?
pass@1 rewards the agent for just getting it right once. The user experiences every run. A task that works 90% of the time is perfectly acceptable in a demo and will fail your customers every single day.
The metric that fixes this is pass^k ("pass hat k"). That's the probability that all k independent attempts at a task succeed.
That was introduced in the tau-bench paper (Sierra, June 2024) for agents.
Per task, with n trials and c successes, you estimate it as:
passk = Etask [ C(c, k) / C(n, k) ]
The maths is brutal. With independent runs, an agent that's successful 90% of the time on a task has a 43% chance of getting 8 in a row.
At 95% it's 66%. At 99% it's 92% (k=8).

The tau-bench authors saw this in practice. Function-calling agents like gpt-4o solved under half the tasks and were "quite inconsistent (pass^8 <25% in retail)".[1]
Princeton's 2026 results throw a spanner in the works for procurement. The smaller models actually often performed as well as or better than the larger ones in terms of consistency.[2] A bigger model is not a consistency strategy.
Does a Temperature of 0 Make an AI Agent Deterministic?
No. Temperature 0 doesn't guarantee identical outputs, and agents have other sources of variance.
Thinking Machines Lab showed in September 2025 that Qwen3-235B at temperature 0 produced 80 unique completions out of 1,000 identical requests. The primary reason for this was server load causing the batch sizes to vary, combined with inference kernels that are not batch-invariant. With batch-invariant kernels, all 1,000 matched.
Most teams don't have any control over their provider's kernels.
Add retrieval results that change, tools that time out, clocks and records that other users edit. The agent won't be deterministic in production. That's perfectly fine.
The test must be. Fixed scenarios, fixed pass criteria and checks against system state give you a deterministic measurement of a non-deterministic system. That's the fundamental idea underpinning all of the stuff that follows.
The Reliability Envelope: How to Test AI Reliability
The Reliability Envelope is a test design that runs every critical agent task through the four rings of Repeat, Rephrase, Break and Bound. A task only passes if it passes in all four rings.
A single good run is a sample. The envelope is the release decision.

| Ring | Question It Answers | How to Run It | Pass Metric | Trust Principle |
|---|---|---|---|---|
| 1. Repeat | Does it succeed every time on the same input? | Run each task k times with identical inputs and fixtures | pass^k per task | Evidence: every run leaves a record |
| 2. Rephrase | Does it survive inputs that mean the same thing? | Paraphrases, typos, reordered details, different personas and registers | pass^k across all variants | Continuous system testing |
| 3. Break | Does it recover when tools misbehave? | Inject timeouts, rate limits, partial responses, schema drift, stale data | Completion under faults vs fault-free | Continuous testing against failures and attacks |
| 4. Bound | When it fails, does it fail inside its limits? | Assert no out-of-scope action, correct escalation, clean stop on command | Zero tolerance: any violation fails the release | Deployer-defined authority; containment |
Ring 1: Repeat
Run the same task with the same inputs and same fixtures k times. Choose k based on the cost of a failure, not on habit. A flow that's moving money or changing customer records is going to warrant a higher k than one that's just producing a summary for a human to look at.
Ring 2: Rephrase
Users will never actually type your test case. You need to build a set of variants for each task that say the same thing but are phrased differently. The evidence that this makes a difference is quite clear.
Princeton found that sensitivity to surface-level paraphrasing varies substantially from model to model.[2]
The ReliabilityBench preprint (arXiv, January 2026) cites one model whose pass rate dropped from 96.9% to 88.1% under input perturbations.
Keep part of the variant set locked away and not seen by the person who is tuning the prompt; otherwise, you'll end up fitting too closely to it.
Ring 3: Break
Inject the faults that your tools actually produce. The evidence is mixed, and that's the point.
Princeton report found that agents did quite well with real technical faults.[2]
A NeurIPS 2026 study of tool-calling agents under realistic MCP faults found the opposite in their setup. Faults drawn from 57,240 real issue and pull request threads led to a 60% drop in task completion scores. Their 120B model scored higher than the 20B model without the faults, but completed fewer tasks than it with them.
The benchmarks disagree, so test with your own fault library, which you build from your own incident history.
Ring 4: Bound
Reliability is not only about how often an agent fails. It's about how badly it fails. Princeton's definition of this is 'safety', i.e., is the severity of the errors bounded?[3]
In the envelope, 'Bound' is a hard wall. On every Repeat, Rephrase and Break run, the agent must never go outside the authority you've defined.
If it's stuck, it must either escalate or stop, and an authorized person must be able to pause it in the middle of the task. One breach and you fail the release regardless of the pass rate.
How Do You Assess AI Reliability Without the Agent Assessing Itself?
Score outcomes against systems of record and absolutely never let the model under test grade its own work. That's our first trust principle (model diversity) applied to evals.
In practice, that means three rules:
- Check the state, not the narration. Did the refund row appear in the database? Did the ticket change state? Did the API get the correct call? The agent's own account of what it did is just a claim, not evidence.
- Use a different judge. Where a criterion is one that requires judgment, use a judge from a different model family than the agent (one that's been calibrated on human-labeled data). A model judging its own output is going to inherit its own blind spots.
- Don't have 'unable to verify' in the pass rate. If a check can't run, then that run is not a pass. Report it as a gap next to the pass rate so a missing check doesn't in any way appear to be a pass.
Each run should leave behind human-readable, tamper-evident evidence that ends up in your own control plane: the inputs, the variant, the injected fault, the actions taken, the state checked and the verdict. That's what turns an AI reliability number into something that an auditor, a risk owner or a customer can actually trust.
What Does an AI Reliability Release Gate Look Like?
An AI reliability gate will block a release unless every critical task has passed the full envelope with evidence. Use this checklist.
- Critical tasks listed and tiered by the cost of failure
- k set per tier (higher for irreversible actions)
- Rephrase set per task, with a sealed holdout the prompt authors never see
- Fault library drawn from your own incidents and tool error logs
- Bound rules written by the deploying team, not inferred from the model
- Outcomes checked against systems of record; judges from a different model family
- Unverifiable results reported separately, never counted as passes
- Re-run the envelope on every change to the model, prompt, tool or retrieval source and on a schedule (as the vendors will be updating their hosted models)
- The same envelope run on every model in your fallback chain so no single model is your only route to a working outcome
The AI Reliability Maturity Model

| Level | What You Measure | What It Catches | What It Misses |
|---|---|---|---|
| 0. Demo | One happy-path run | Broken builds | Almost everything |
| 1. Averaged evals | Mean score over a test set | Capability regressions | Run-to-run variance |
| 2. Repeat | pass^k on fixed inputs | Inconsistency | Phrasing and fault brittleness |
| 3. Envelope | pass^k across Repeat, Rephrase, Break | Real-world brittleness | Severity of failure |
| 4. Gated and evidenced | Full envelope plus Bound, in CI, with sealed evidence | Unsafe failures before release | What production teaches you next, so feed it back |
Where Does the Reliability Envelope Actually Break Down?
It has limits, and naming those limits makes it far stronger. Five edge cases come up in every team that adopts it.
- Open-ended tasks. If you're producing a summary or a draft, you want the wording to be varied. You apply consistency to the outcome criteria (facts present, policy followed, no invented numbers), not to the text.
- Cost. The envelope multiplies out to k times variants times faults. Tier it. Full envelope on the critical tasks, repeat only on the low-risk ones, and run everything in parallel so the gate stays fast.
- Correlated failures. pass^k assumes independent runs. Real failures cluster around a phrasing or a fault, which is exactly why Rephrase and Break exist as separate rings.
- In-house builds. Independence doesn't require a vendor. It requires a verifier who doesn't share the agent's model and whose pass criteria are owned by someone other than the person who's tuning the prompt.
- Owning your evidence. If your evidence exists only in a vendor's dashboard, you don't own it. Keep your run evidence in systems that you own.
Where Do AI Observability and Evaluation Fit In?
Observability tells you how the agent behaved post-release. The envelope tells you how it will behave pre-release. You need both, and most teams only have the first.
LangChain's State of Agent Engineering survey (1,340 respondents, fielded late 2025) found that 89% of organizations had some degree of agent observability. Only 52.4% ran offline evaluations, and 37.3% ran online evaluations. Quality was the top barrier to production (32%), and that's where the reliability issues are.
You can see the failures in production, but you can't predict them prior to release.
The two halves meet in a loop. The production traces and incidents get fed into your rephrase and break libraries. The envelope then proves the fix is still valid prior to the next release.
How TestMu AI Applies the Reliability Envelope
At TestMu AI we think of the 'Assure' in the Observe, Eval and Assure cycle as happening prior to release and is based on an independent verification layer which produces evidence rather than opinion.
- Agent Assurance (delivered via Rook) runs an agent against a suite built from your repo, PRD or docs. It checks the effect on tools, files and APIs. Every verdict is Pass, Fail, or Unable to Verify. Unverifiable results are excluded from the pass rate and appear as an assurance gap. Adversarial cases (prompt injection, tool misuse, etc.) are generated by default, and gate recipes are available for GitHub Actions, Jenkins and Argo CD.
- Agent Testing runs simulated users against the chat, voice, IVR and video agents prior to each deployment, which is where the Rephrase ring does the majority of its work.
- Kane CLI and KaneAI cover the human-to-software flows that an agent interacts with, and HyperExecute will run envelope-sized suites in parallel.
We hold ourselves to the same test that we're asking you to apply. Our verifier is not your agent's model.
Agent Assurance keeps the per-criterion evidence as plain old files on disk (in your hands as you run it) and produces one JSON report per run. Ask any vendor (including us) the same questions.

What Happens Next: Our Prediction
The direction is clear. Leaderboards such as the Agent Arena Index are incorporating reliability signals. Princeton has turned consistency, robustness, predictability and safety into named metrics.
In Europe, Article 15 of the EU AI Act requires high-risk systems to be "as resilient as possible regarding errors, faults or inconsistencies".
The Digital Omnibus on AI has moved the high-risk obligations for Annex III systems to 2nd December 2027.
Our prediction: by the end of 2027, pass^k will be a metric that sits alongside accuracy in agent leaderboards and in the enterprise AI vendor questionnaires. Any team that can't demonstrate one per release is going to see their agent rollouts hamstrung by risk review.
Conclusion: AI Reliability Is Earned per Release
AI reliability isn't a property of the model. It's a property of a particular agent on particular tasks under particular conditions at a particular release.
Measure it like any other property. Run each critical task repeatedly. Put it the way users will word it.
Break the tools it relies on. Bound what it's allowed to do when it fails. Ship on evidence, not on your best run.
Shipping agents that other agents use? Rook by TestMu AI does Agent Assurance: agents testing agents before release, with no human in the loop.
npm install -g @testmuai/rook@latest command, or for a full walkthrough of the product.Author
Mayank Bhola is Co-Founder and Head of Products at TestMu AI, where he leads the entire product portfolio across KaneAI, Kane CLI, HyperExecute, SmartUI, the Real Device Cloud, Accessibility, and other software testing product lines. As an early Lead Architect he designed and built the company's flagship Tunnel technology from scratch, created the React-based automation platform, and architected the data-intensive pipelines and FAAS services that scale it. He brings more than 10 years of experience in software development and product engineering, with earlier roles as Head of Technology at Juggernaut Books and Senior Software Engineer at PressPlay TV and Zomato. Mayank holds a B.Tech in Computer Engineering from JIIT Noida.
Reviewer
Maneesh Sharma is Chief Operating Officer at TestMu AI, where he leads the company's revenue functions across go-to-market, sales, RevOps, customer success, and the partner ecosystem, driving its global expansion. He brings over 28 years of experience across product, go-to-market, and business operations. Before TestMu AI he was Managing Director and General Manager for GitHub across India and APAC, Head of Partner Sales and Ecosystem at Adobe, Chief Revenue Officer at WizIQ, and Head of Products and Solutions at SAP India. Maneesh holds an MBA from IIM Bangalore and a B.E. in Electronics and Communications from NIT Surat.
AI Reliability FAQs
Did you find this page helpful?
More Related Blogs
TestMu AI forEnterprise
Get access to solutions built on Enterprise
grade security, privacy, & compliance
- Advanced access controls
- Advanced data retention rules
- Advanced Local Testing
- Premium Support options
- Early access to beta features
- Private Slack Channel
- Unlimited Manual Accessibility DevTools Tests
















