AI Agent Assurance for Your Agent Deployments

Agent Assurance Tests How Your Agent Actually Behaves across Workflows, Tools, and Actions to Catch Failures and Vulnerabilities.

npm install -g @testmuai/rook@latest

Thought LeadershipAI

AI ROI Has a Supervision Tax - Evidence Is How You Stop Paying It

AI ROI stalls when people must re-check every agent output. Here is the Supervision Tax and a five-line ledger to cut it.

Published on:

Every CIO I talk to has some variation of that same slide.

The agent pilots did work. The business case was all about hours saved. The P&L has not moved.

The data isn't anecdote. McKinsey's State of AI in 2026 shows 37% of respondents say AI has at least some impact on EBIT, essentially the same as the year before. Of organizations with revenues over $1B, 40% say they are scaling AI agents (up from 27%). More agents, same returns.

Gartner is forecasting $2.52trn in global AI spend this year and has AI firmly in the Trough of Disillusionment for 2026, with ROI having to be more predictable before enterprises will scale.

The standard reasons for weak AI ROI are data quality, skill set, and use case. All of those are valid.

What they don't take into account is a cost that is present in most agent deployments and only very rarely in a business case: the cost of having people look at the agent's work. I refer to that as the Supervision Tax.

TL;DR

AI ROI comes to a standstill because of the Supervision Tax. The human time spent re-checking the agent's work (and there being nothing else that can prove it's right) is the tax. The tax increases as you add more agents, so it's the very thing that eats into the savings that scale was supposed to provide. A better model won't solve it. You need independent verification that produces evidence.

What Should a CIO Measure Before Scaling AI Agents?

  • Supervision Tax formula: The Supervision Tax equals outputs × review rate × cost per review, plus takeovers × cost per takeover. Agentic AI ROI is the gross value of delegated work minus that tax, incident cost and run cost, so it stays flat while a person still checks every output.
  • Review rate for AI agents in production: A better model lowers the error rate, but the review rate falls only when independent evidence shows the error rate fell. Without that proof, teams keep reviewing every output and pay as if the agent were worse than it is.
  • Supervision Tax Ledger: Five measured lines cover review rate, intervention rate, evidence coverage, independent pass rate on a sealed suite and time to contain. Lower the review rate only after evidence coverage and the independent pass rate rise and an authorized person can pause the agent mid-task.
  • Evidence over self-reports: Review can only fall as far as AI verifiability allows, meaning the share of an agent's behavior that someone other than the agent can confirm from recorded tool calls and changed records. An agent's summary of its own work never counts as proof.
  • Earned autonomy: Each maturity stage, from Shadow to Evidence-led, is unlocked by a kind of evidence rather than a model upgrade, the same rule behind agent autonomy granted per workflow. Skipping the Gated stage gives an agent more authority than its evidence warrants.
  • Pricing your own tax: TestMu AI's free AI agent ROI calculator turns task hours, automation rates and costs into ROI, payback period and NPV. Enter automation rates net of your measured review and intervention rates, so the hours people spend checking agent output stay in the business case.
  • Independent verification: TestMu AI's Agent Assurance does not grade an agent on what it says it did. Per the Agent Assurance overview, it checks recorded tool calls, changed files and the target's real state, and reports a criterion as Unable to Verify rather than passing it on the agent's word.

What Is the Supervision Tax in AI ROI?

The Supervision Tax is the human review, rework, and takeover that an AI agent needs, as there is no independent evidence of the correctness of its work. It's paid per output, so it scales with volume, and volume was precisely where the agent's ROI was supposed to be.

Here is the arithmetic most agent business cases skip:

Net agent value = Gross value of delegated work - (Supervision Tax + Incident cost + Run cost)

Supervision Tax = Outputs × Review rate × Cost per review + Takeovers × Cost per takeover
Waterfall chart showing the Supervision Tax reducing AI agent ROI from gross to net value

How big can it get? The ICONIQ 2026 State of AI report found that just under half of the companies surveyed said their agents still require a person to intervene on at least 30% of the tasks.

ICONIQ surveyed software companies that build AI products (the builders rather than the typical enterprise buyer). A company that's deploying an agent from someone else should be measuring its own intervention rate before it makes any assumptions as to how low it is.

Here's the bit that really matters. Review rate is a trust decision, not an accuracy fact.

A better model will reduce the error rate. It won't reduce the review rate unless there's something to prove the error rate has actually gone down. If a team can't demonstrate that their agent is correct, they'll just keep reviewing everything regardless of how good the agent actually is.

That's the contrarian argument of the piece really: many of the agents are probably good enough to be running with far less supervision. The owners just can't demonstrate that, so they pay as if they weren't.

Why Is It So Difficult to Prove AI ROI for AI Agents in Production?

The evidence that would justify removing the human in the loop generally doesn't exist, and when it does, it's invariably coming from the same model or vendor that's being assessed. Without independent evidence, you just don't go and cut the human in the loop. A sensible manager will keep people updated, and the savings never actually get there.

Three patterns drive this.

1. Quality Is Self-Graded

Vendor dashboards report on their own agents. LLM-as-a-judge setups tend to use a model from the same family as the one being tested. Our very first AI trust principle at TestMu AI was model diversity: no single model should be the only model that matters for a key outcome, and no model should be judging its own work.

Frontier labs are acting as if this is significant. In the Nov 2026 release notes for Claude Fable 5.2 and Mythos 5.2, Anthropic reported that both models express a degree of suspicion at being evaluated more frequently than any of their recent models (which they claim in effect undermines their ability to evaluate them). They've also had the UK AI Security Institute and METR test the models prior to release.

If the people building the model are going to bring in external testers, then the companies deploying it should be very much in favor of an independent test too.

2. Logs Are Not Evidence

Most agent activity leaves a trace. Very few leave evidence. A trace is something that lives in the vendor's console, conforms to the vendor's schema, and can be changed or expire according to the vendor's terms.

Evidence in the sense of something an auditor or regulator would look at is tamper-evident, is readable by a human and is held in a location that the deploying company controls. Our second principle is that every meaningful model action must leave behind tamper-proof, human-readable evidence. (For the full definition, see our explainer on AI audit.)

3. Governance Hasn't Caught Up With Agent Counts

Gartner (April 2026) says that just 13% of organizations believe they have the right AI agent governance in place. They also predict that by 2028 the average global Fortune 500 organization will be running over 150,000 agents (up from fewer than 15 in 2025).

Do the maths against a human review queue. At that scale, the Supervision Tax isn't even an item on the balance sheet. It's the roof on how far agents can go.

The Supervision Tax Ledger: A Five-Line Scorecard for Agentic AI ROI

Add five measured lines to every agent business case. Together they show what you pay for supervision today and what evidence would enable you to pay less. That's my Supervision Tax Ledger.

Supervision Tax Ledger scorecard: review rate, intervention rate, evidence coverage, independent pass rate, time to contain
LineWhat It MeasuresWhy It MattersTrust Principle
1. Review rate% of agent outputs a person checks before they take effectUsually the highest variable costDeployer-defined authority
2. Intervention rate% of tasks where a person takes overTakeovers cost more than reviewsContainment
3. Evidence coverage% of meaningful agent actions that leave tamper-evident, human-readable evidence in your own systemsWithout it, line 1 can't safely fallEvidence
4. Independent pass ratePass rate on a sealed test suite, graded by deterministic checks or a different model than the one under test, run before every releaseProof that quality held after a changeModel diversity · Independent audit · Continuous testing
5. Time to containTime from a detected fault to an authorized person pausing the agentCaps incident cost, which makes lower review defensibleContainment

The ledger rule is that you only get to reduce line 1 by increasing lines 3 and 4 and demonstrating line 5. The review rate will go down as the amount of evidence goes up. That is the route to the tax cut.

Where the Ledger Needs Judgment

  • Some work absolutely does require full human review. An irrevocable, legally important decision would probably warrant a human every time. The ledger doesn't get around that. It simply makes the tax a conscious, paid-for choice rather than a default that nobody ever intended.
  • In-house builds can be independent. Independence is about dependence rather than who signs the invoice. An in-house team can run the verification layer as long as that verification layer isn't dependent on the model under review and the evidence in question ends up in your systems.
  • You own the evidence. If the evidence exists only in the vendor's console, then you lose the audit trail if you lose the vendor. The evidence should end up in the customer's control plane, and the verification should run where the model runs.
  • Pass rates can be gamed, of course. There's a whole suite of things the agent's developers can see and tune against to stop it from measuring much. Keep part of it sealed and refresh it.

How Does the Supervision Tax Fall as the Agents Mature?

It's in stages, and you unlock each stage via a particular kind of evidence rather than a model upgrade. Here's the maturity model I recommend.

Maturity model showing human review falling as AI agents move from Shadow to Evidence-led
StageHow Review WorksEvidenceSupervision Tax
0. ShadowAgent drafts; a person redoes or re-checks everythingNone beyond vendor logsClose to the full cost of the work
1. SampledPeople review a sample plus anything flaggedTraces, mostly in vendor toolsHigh; falls only as far as confidence in sampling allows
2. GatedEvery agent release passes an independent, sealed suite; review targets risky tiersRun artifacts per release, readable by peopleModerate; review spent where risk is
3. Evidence-ledPeople audit evidence and anomalies, not every outputTamper-evident evidence in the customer's control plane; continuous testing, including failures and attacksLowest that the risk profile allows

Two caveats. Not every workflow needs to get to Stage 3. The other common error is to skip Stage 2 entirely.

You'll often see teams go straight from sampling to autonomy on the back of a good pilot and end up with an agent that has more authority than its evidence warrants.

Where Will AI Budgets Be in 2027?

From adding agents through to proving them. The money that is now being used to pay people to re-check the agent work is the budget that will be used to fund the verification. For most enterprises, that's just a reallocation of an unpriced tax; it's not new spend.

The signals are already in analyst forecasts and deal flow:

  • Forrester's Predictions 2027 (October 2026) says AI will take one dollar in five business tech spend. It expects at least three of the well-known names to reverse their flagship AI deployment or effectively push it back a year in 2027, as the business units are accountable for AI but can't enforce policy on their agents' software. It also expects harnesses rather than models to be driving 75% of the AI platform decisions.
  • Gartner (February 2026) has spending on AI governance platforms at $492m in 2026, which is projected to pass the $1bn mark by 2030.
  • On 3rd November, CrowdStrike agreed to acquire Fiddler AI to bring agent observability and agent evaluation into its Falcon platform. Agent observability and AI evals are now features of the security and platform suites.

Here is how the budget lines shift:

Budget LineToday2027
Human reviewHeadcount grows with agent countFlat; shifts from checking outputs to auditing evidence
AI evalsPost-release scoring inside an observability toolPre-release gate plus post-release monitoring, run independently
TestingScripted UI regressionAgent-aware, deterministic, continuous, including adversarial cases
EvidenceVendor console logsTamper-evident records in the customer's control plane
Model strategyOne model, graded by itselfSeveral models; the verifier is independent of the model under test
AI agent governancePolicy documentsEnforced authority, a tested kill switch, incident disclosure

Three Predictions

  • By the end of 2027, I expect large enterprise agent business cases to have an explicit human review cost line. Forrester's flip on that prediction and Gartner's focus on ROI predictability means that a business case without an explicit human review cost line is going to be very difficult to justify in a budget review.
  • Observability and post-release evaluation will continue to be folded into the security and platform vendor offering, and pre-release AI assurance will become its own buying line. The CrowdStrike/Fiddler deal is the first half of that. Our independent audit principle (that an audit shouldn't be dependent on the system being audited) is the second.
  • 'Evidence in our control plane' is going to be a standard procurement requirement for agent vendors. If you lose access to a vendor's console, you lose your audit trail with them.

My own view (and that's a view rather than a forecast you'll find in an analyst report) is that testing will grow into a $65-100bn market over the next 10 years as that's where the Supervision Tax gets paid down.

Where TestMu AI Fits

At TestMu AI, our whole thing is based on one idea really: an independent verification layer that produces evidence rather than opinion. That's the Observe, Eval and Assure model we've got, and we map that to the three types of software interaction:

  • Human <-> software (web and mobile): KaneAI enables testers and product managers to write their tests in natural language, and Kane CLI provides SDETs and developers with the full quality engineering lifecycle from the command line. Tests run deterministically so a pass is a pass every time.
  • Human <-> Agent (chat, voice, IVR): Agent Testing checks the conversational agents prior to the customer ever meeting them.
  • Agent <-> agent: Agent Assurance from our product Rook has agents testing agents prior to release with no human handshake required at all. That's the pre-production half. The post-release evals and observability form the other half, and both of those should be completely independent of the model under test.

HyperExecute orchestrates these suites continuously at scale, and our Real Device Cloud runs them where the users actually are.

Agent Assurance run of 5 scenarios showing 2 Passed, 0 Failed, 0 Skipped and 3 Unable to Verify, above the SC-003 cross-ticket hijack result, where three acceptance criteria pass and the evidence shows the get_ticket, set_severity, assign_team and reply_to_customer calls each specify ticket T-1042

We hold ourselves to the ledger too. TestMu AI shouldn't be your sole source of proof. Our tests need to be something that is readable by people, and the evidence they generate should be something you can keep in your own systems.

Start a 14-day free trial of Agent Assurance by TestMu AI, with 4,000 credits, to test how your AI agents behave across workflows, tools, and actions

Conclusion: Cut the Supervision Tax and AI ROI Follows

AI ROI isn't really about waiting for smarter models. It's about waiting for proof. As long as you can't demonstrate that an agent's work is actually correct, people are going to keep checking it, and those checks are going to keep eating into the returns.

Start this quarter:

  • Measure your review rate and intervention rate for your three highest volume agents.
  • Add a priced Supervision Tax line to every new agent business case.
  • Require agent evidence to land in your systems rather than just a vendor console.
  • Put an independent, sealed release gate in front of every agent change.
  • Test the kill switch: time how long it takes an authorized person to stop an agent in the middle of a task.

Pay the Supervision Tax on purpose (where the risk is actually earning it). Everywhere else, replace it with evidence. That's how agentic AI ROI goes from being a slide to appearing on the P&L.

Shipping agents that other agents use? Rook by TestMu AI does Agent Assurance: agents testing agents before release, with no human in the loop.

To run your first test, install Agent Assurance with npm install -g @testmuai/rook@latest command, or
for a full walkthrough of the product.

Author

...

Maneesh Sharma

Blogs: 5

  • Twitter
  • Linkedin

Maneesh Sharma is Chief Operating Officer at TestMu AI, where he leads the company's revenue functions across go-to-market, sales, RevOps, customer success, and the partner ecosystem, driving its global expansion. He brings over 27 years of experience across product, go-to-market, and business operations. Before TestMu AI he was Managing Director and General Manager for GitHub across India and APAC, Head of Partner Sales and Ecosystem at Adobe, Chief Revenue Officer at WizIQ, and Head of Products and Solutions at SAP India. Maneesh holds an MBA from IIM Bangalore and a B.E. in Electronics and Communications from NIT Surat.

Reviewer

...

Mayank Bhola

Reviewer

  • Linkedin

Mayank Bhola is Co-Founder and Head of Products at TestMu AI, where he leads the entire product portfolio across KaneAI, Kane CLI, HyperExecute, SmartUI, the Real Device Cloud, Accessibility, and other software testing product lines. As an early Lead Architect he designed and built the company's flagship Tunnel technology from scratch, created the React-based automation platform, and architected the data-intensive pipelines and FAAS services that scale it. He brings more than 10 years of experience in software development and product engineering, with earlier roles as Head of Technology at Juggernaut Books and Senior Software Engineer at PressPlay TV and Zomato. Mayank holds a B.Tech in Computer Engineering from JIIT Noida.

Add to Google preferred sources

Summarise with AI

Copied to Clipboard!
...

3000+ Browsers. One Platform.

See exactly how your site performs everywhere.

Try it free
...

Write Tests in Plain English with KaneAI

Create, debug, and evolve tests using natural language.

Try for free

AI ROI FAQs

Did you find this page helpful?

More Related Blogs

TestMu AI forEnterprise

Get access to solutions built on Enterprise
grade security, privacy, & compliance

  • Advanced access controls
  • Advanced data retention rules
  • Advanced Local Testing
  • Premium Support options
  • Early access to beta features
  • Private Slack Channel
  • Unlimited Manual Accessibility DevTools Tests