AI Agent Assurance for Your Agent Deployments

Agent Assurance Tests How Your Agent Actually Behaves across Workflows, Tools, and Actions to Catch Failures and Vulnerabilities.

npm install -g @testmuai/rook@latest

Thought LeadershipAI

AI Governance Tools Track Your Agents - Which Ones Prove They Work?

AI governance tools can list every agent you run. Few can prove what those agents did. A CIO's 4-layer Governance Evidence Stack and 10-question scorecard.

Published on:

The AI governance tools market is now getting serious money. Gartner forecasts AI governance platform spend will hit $492m in 2026 and pass the $1bn mark by 2030.

Microsoft reports that over 50 million agents have been registered in Agent 365 since May.

LangChain raised $300m last week to 'run, improve and govern agents'.

The registry problem is being solved. The proof problem is not.

On 16th November, Gartner published a survey of 298 infrastructure and operations leaders. 95% had at least one agent-related incident over the past 12 months. 70% feel their agents are operating within safe boundaries. Only 15% have automatic circuit breakers.

That combination of high confidence, very thin controls, and widespread incidents is what I call the Confidence-Evidence Gap. The vast majority of AI governance tools on the market today are designed to help you achieve the first of those three numbers. Very few actually help you close the gap to the third.

The Confidence-Evidence Gap in AI agent governance, Gartner 2026 survey data

This is for the CIO and the founder deciding where the governance budget goes in 2027. It's a market map, a named model and ten questions you can ask any vendor (including us).

TL;DR

AI governance tools are great at telling you what agents are out there and what they're permitted to do. Most of them can't actually prove what those agents did. Before you go and buy one, test it across the four layers (Inventory, Policy, Evidence, Attestation) and spend your money where the evidence is thin, i.e., independent evidence that was generated prior to release.

Which AI Governance Tools Prove What an Agent Actually Did?

  • Agent registries: Inventory layer. Proves the agent behaved: No. An agent registry records which agents and models an organization runs and who owns each one, which answers what exists but not whether any agent did the right thing on a given run.
  • Governance and GRC platforms: Policy layer. Proves the policy held: No. Governance platforms and GRC workflows record what each agent may access and do, so they hold intent; whether a policy held under real inputs and attacks needs evidence from a separate check.
  • Independent pre-release testing: Evidence layer. Can block a release: Yes. Testing that runs outside the model under review, gives the same result on the same build and confirms outcomes in systems of record produces the proof that the Attestation layer depends on.
  • Agent observability: Evidence layer, after release. Stops a build before it ships: No. Agent observability shows that an agent drifted in production, so it complements a pre-release gate rather than replacing one.
  • AI evals: Evidence layer. Counts as proof when self-graded: No. AI evals score a model or agent's outputs, and they count as evidence only when the grader is independent of the model family that runs the agent being graded.
  • ISO 42001 certification: Attestation layer. Produces its own evidence: No. ISO/IEC 42001 certification audits an organization's AI management system, and a certifier can only attest to evidence that already exists, so attestation is as strong as the Evidence layer beneath it.
  • TestMu AI's Agent Assurance: Evidence layer, before release. Takes the agent's own report as proof: No. Per the Agent Assurance overview, it checks recorded tool calls, changed files and the target's real state, and reports a criterion as Unable to Verify rather than passing it on the agent's word.
  • AI agent governance: AI agent governance sets owners, permissions and release rules for agents that act, and those rules hold up with an auditor only when the Evidence layer shows they held on real runs.

What Are AI Governance Tools and What Do They Actually Do?

AI governance tools are software that catalogs AI systems, attaches policies and owners to them, and records the decisions for regulators and auditors. The good ones now cover the agents as well as the models. What they predominantly record is intent (i.e., what should happen) rather than evidence of what actually happened.

Look at page one of Google for "AI governance tools", and you'll see all the feature lists: inventory, risk assessment, policy workflows, monitoring dashboards, compliance mapping. All of that's very useful.

But the regulator, the auditor, or the customer's procurement team is going to eventually ask a different question. They're going to want to know what the agent actually did and who looked at it.

That question divides the category in two. Some tools will record what you intended. Others will produce evidence of what actually happened.

The buyers who blur the two together end up with a lovely register and nothing to hand an auditor.

Why Isn't an Agent Inventory Sufficient for AI Agent Governance?

An inventory proves an agent exists; it doesn't prove that the agent behaves. Registering 50m agents is a huge scale achievement, but registration answers the question "what do we have?" rather than "did it do the right thing on Tuesday?"

The survey data agrees. Deloitte found in August 2026 that 74% of companies plan to deploy agentic AI within two years, while only 21% report a mature governance model for autonomous agents.

KPMG's Q3 2026 pulse of 314 US leaders found confidence in governance rising to 73% - while the share building controls into the agents themselves fell to 30%, from 43% two quarters earlier.

Confidence is up. Built-in controls are down. That's the Confidence/Evidence Gap for the quarter.

There's also a rather new problem that no register can fix. Anthropic's November launch post for its latest models notes that the models themselves voice suspicion that they're being evaluated more often than any of its recent models (the company puts it that way), which in turn makes it very difficult to assess how they'd behave in the real world.

If the system in question can detect that it's being observed, then a self-reported compliance figure is of absolutely no value. You need evidence that's been gathered by something that the model has no control over.

The Governance Evidence Stack: A Four-Layer Model for AI Governance Tools

The Governance Evidence Stack is a four-layer model for judging AI governance tools on what each layer can prove: Inventory (what's there), Policy (what's permitted), Evidence (what actually happened, independently verifiable), and Attestation (who signs off). Each layer is dependent on the one below it.

You can't have Attestation without Evidence. Attestation is just an opinion with a letterhead.

The Governance Evidence Stack: four layers of AI governance tools, from agent inventory to independent attestation
LayerQuestion It AnswersTypical ToolsWhat It Cannot Prove
1. InventoryWhat agents and models do we run, and who owns them?Agent registries, AI asset discovery, shadow-AI detectionWhether any of them behave correctly
2. PolicyWhat is each agent allowed to access and do?Governance platforms, GRC workflows, identity and permissions, guardrailsWhether the policy held under real inputs and attacks
3. EvidenceWhat did the agent actually do, and did it pass, before and after release?Independent testing and verification, agent observability, evalsNothing - but only if it is independent of the model under review
4. AttestationWill a qualified third party sign off?Big Four AI assurance, ISO/IEC 42001 certification bodies, notified bodiesAnything Layer 3 did not capture

Three rules in practice make the stack work.

Rule 1: Evidence is the load-bearing layer. Policy without evidence is a promise. Attestation without evidence is a guess. If you're going to fund just one layer properly in 2027, fund layer 3.

Rule 2: Evidence must be independent of the model under review. That's the first of the AI trust principles we have at TestMu AI. No single model is the sole dependency for an important outcome, and no model is going to check its own work. An evaluation suite where the same model family is assessing its own agent is just layer 2 in disguise as layer 3.

Rule 3: Evidence must end up in your control plane. If your proof lives only in a vendor's dashboard, you do not own it. Evidence should be tamper-evident, readable by a human, and exportable to systems you control - so an auditor, an insurer or a regulator can read it without asking your vendor's permission.

Two edge cases matter.

In-house builds. You can build layer 3 yourself. The test isn't about who wrote the code.

The test is whether the checker is independent of the system it's checking and whether the result is deterministic and repeatable. A platform team that checks agents against real systems of record passes. An agent that writes its own success report fails.

Owning your own evidence. Some teams are concerned that if you send your evidence to an external verifier, you're then reliant on them. Rule 3 addresses that: the verifier will produce the evidence, but the evidence is yours.

That also applies to continuity of access to your own audit trail. It should not be something that a vendor can take away or that a policy you have no control over can take away after you've deployed.

Where Is AI Governance Tool Spending Heading?

Money's at the bottom (inventory), and the top (attestation) of the stack; the middle (independent evidence) is still underbuilt. That's where I expect the next budget shift to occur.

Market map of AI governance tools by what they can prove and when

Look at the last four weeks:

  • Inventory has scaled. Agent 365's registry now includes agents from a long list of platforms (Salesforce Agentforce, Amazon AgentCore, Google Vertex AI, Databricks, Snowflake and others) as per the Microsoft Ignite 2026 update.
  • Observability consolidated. CrowdStrike agreed on 3 November to acquire Fiddler AI and repackage it as Falcon Observe (including pre-deployment evaluation). LangChain's raise was being pitched around running, improving and governing agents.
  • Attestation professionalized. PwC launched 'Assurance for AI' in June 2025. Deloitte broadened its AI controls and assurance services in August 2026. EY US announced Assurance Forward Deployed Engineers in November 2026 as part of a $1bn investment in talent and technology. ISO/IEC 42006 (published July 2025) now defines the requirements for the bodies that are going to audit and certify AI management systems. BSI reported in October 2026 that it had issued its 50th ISO/IEC 42001 certificate (up from 11 in all of 2025).

Attestation is clearly increasing. An auditor can only attest to evidence that actually exists, so layer 3 is the bottleneck for everyone above it.

The regulatory timetable is all pointing in the same direction. Under Article 43(2) of the EU AI Act, most of the high-risk systems in Annex III (points 2 to 8) are subject to internal control only (i.e., no notified body is involved). This is as per the Digital Omnibus.

The obligations apply from 2 December 2027. So for the vast majority of high-risk use cases, the provider is going to have to put together its own evidence and be behind it. No one is going to be creating that evidence for you.

The UK is building the profession around this. The government's Trusted Third Party AI Assurance Roadmap (Sept 2025) identifies over 524 UK firms providing AI assurance and £1.01bn of gross value added. It forecasts the market could reach £18.8bn by 2035 if the barriers to AI adoption are removed.

What's the Difference Between AI Governance and AI Assurance?

AI governance sets the rules; AI assurance verifies that the rules have been adhered to. Governance is layers 1 and 2. Assurance is layers 3 and 4.

DimensionAI GovernanceAI Assurance
Core questionWhat should our AI do, and who owns it?Can we prove what it did?
Main outputInventory, policies, risk registerTest evidence, audit trail, attestation
When it runsDesign time and ongoing reviewBefore release (gate) and continuously after
Who buys itCIO, CISO, chief risk officerSame buyers - plus auditors, insurers, regulators who read it
Failure modePolicies nobody testsScores the model grades itself

Pre-production assurance and post-production observability are complementary rather than interchangeable. Observability will tell you that an agent drifted last night. A release gate will tell you that it shouldn't have shipped.

You need both, and they shouldn't share a single point of failure.

How Do You Evaluate AI Governance Tools? The 10-Question Proof Test

Ask every vendor 10 questions. If they say no to any of the first 6 questions, then it's a register, not a source of proof. Use that as your scorecard in RFPs.

  • Independence. Is the check being done by something other than the model (or the model family) that's actually running the agent?
  • Determinism. Will the same test on the same build give the same result? Can you re-run it?
  • Outcome not self-reported. Does it confirm the outcome in the systems (the refund has been issued, the ticket has been closed) rather than just relying on the agent's summary?
  • Release gate. Can it stop a deployment if the evidence is missing or failing?
  • Tamper-evident record. Is every meaningful agent action recorded in a form that would show if it had been altered and that a human could read it?
  • Your control plane. Can you export all the evidence to systems you own in an open format without the vendor's assistance?
  • Adversarial coverage. Does it test for failures and attacks (prompt injection, broken tools, bad data) all the time, rather than just the happy paths?
  • Deployer-defined authority. Does it actually test that the permissions you've set are valid rather than just relying on the model's own assessment?
  • Containment drill. Can it prove that an authorized person can actually pause or stop an agent in the middle of a task? Has that been drilled?
  • Disclosure ready. If anything goes wrong, can you produce a timeline for the affected parties fairly quickly?

Score each 0 (no), 1 (partly) or 2 (yes, demonstrated). Under 12 out of 20, you're buying a register. 16 or more, you're buying evidence.

This scorecard maps directly to the seven AI trust principles we publish at TestMu AI (model diversity, evidence, continuous system testing, deployer-defined authority, independent audit, containment and incident disclosure). A principle you can't test is just a slogan.

Where Does TestMu AI Fit in the AI Governance Stack?

TestMu AI builds layer 3: an independent verification layer which produces evidence rather than opinion. We don't build the agents that we test, and our testing is deterministic (the same test on the same build will always produce the same verdict).

Across the three ways software is used today, that looks like this. For human-to-software flows, teams write natural-language end-to-end tests with KaneAI, or run the full QE cycle from the terminal with Kane CLI. For human-to-agent channels, teams test chat, voice and IVR agents with Agent Testing before customers meet them.

For agent-to-agent workflows, our Agent Assurance product, Rook, acts as an independent verification layer between agents before release. HyperExecute lets you orchestrate those runs at scale.

Agent Assurance run results showing 6 scenarios with 5 passed, 0 failed, 0 skipped and 1 Unable to Verify, above the detail for scenario SC-001, where six criteria pass and two call-count checks are marked Unable to Verify because rook cannot show the tool-call record is complete

We hold ourselves to the same Proof Test. Our evidence should be able to export to your systems, and we should never be the only check on an outcome that actually matters. If a vendor (us included) can't pass questions 1 to 6, then you should keep on shopping.

Start a 14-day free trial of Agent Assurance by TestMu AI, with 4,000 credits, to test how your AI agents behave across workflows, tools, and actions

What Happens Next: Three Predictions for AI Governance Tools

These are my predictions, not certainties, each one based on a trend cited above.

  • By the end of 2027, serious AI governance RFPs will be asking for exportable test evidence rather than just inventories. Basis: the EU Annex III obligations come into force on 2 December 2027 under a self-assessment regime, and the Big Four assurance practices are already scaling. The auditors will be asking for the evidence, and the buyers will be pushing that requirement through to their tools.
  • The governance category will split into 'registers' and 'evidence producers' in two years. Basis: the November funding and M&A wave was overwhelmingly to registries, controls and observability, with Gartner's own data showing only 15% of I&O teams have automatic circuit breakers. That gap is a product opportunity, and someone will definitely step in to fill it.
  • "Who checked?" will be the go-to question in procurement for agents. Basis: On 16 November, the California AG brought proceedings against OpenAI, claiming that during their testing of autonomous agents, OpenAI had lost control at least 80 times. Whatever the result, testing practice is now something that regulators will be looking at. Boards will be keen to have their own answer to that in place.

I've been arguing for some time that testing will become a $65-100bn market over the next 10 years. This is why moves like this are so important: AI governance tools create the demand for proof, and verification provides it.

Conclusion: Buy Proof, Not Just a Register

AI governance tools are necessary, and the inventory layer is in a good state. The Confidence/Evidence Gap, however, will not close with more dashboards. It closes when every important agent action has left behind evidence that an independent check has been performed, that the human can read and that you own.

Use the Governance Evidence Stack to see where your spend is. Run the Proof Test on every vendor. Put your next dollar where the stack is thinnest, i.e., independent evidence before release.

Further reading from our glossary: AI audit, AI TRiSM, AI verifiability and agent harness.

Shipping agents that other agents use? Rook by TestMu AI does Agent Assurance: agents testing agents before release, with no human in the loop.

To run your first test, install Agent Assurance with npm install -g @testmuai/rook@latest command, or
for a full walkthrough of the product.

Author

...

Maneesh Sharma

Blogs: 5

  • Twitter
  • Linkedin

Maneesh Sharma is Chief Operating Officer at TestMu AI, where he leads the company's revenue functions across go-to-market, sales, RevOps, customer success, and the partner ecosystem, driving its global expansion. He brings over 27 years of experience across product, go-to-market, and business operations. Before TestMu AI he was Managing Director and General Manager for GitHub across India and APAC, Head of Partner Sales and Ecosystem at Adobe, Chief Revenue Officer at WizIQ, and Head of Products and Solutions at SAP India. Maneesh holds an MBA from IIM Bangalore and a B.E. in Electronics and Communications from NIT Surat.

Reviewer

...

Mudit Singh

Reviewer

  • Linkedin

Mudit Singh is Co-Founder and Head of Growth at TestMu AI, and a member of the founding team that has grown the platform to 2.8M+ users across 132 countries. He drives growth and go-to-market across the full product portfolio, including KaneAI, Kane CLI, the Real Device Cloud, SmartUI, accessibility testing, and HyperExecute. Over more than eight years on the founding team he has helped build and bring these testing products to market from scratch, scaling the company from cross-browser testing into a full-stack agentic AI quality engineering cloud backed by Sequoia, Qualcomm Ventures, and Premji Invest. He brings over 15 years of experience building and marketing software products and holds a B.Tech in Computer Science from Jaypee Institute of Information Technology.

Add to Google preferred sources

Summarise with AI

Copied to Clipboard!
...

3000+ Browsers. One Platform.

See exactly how your site performs everywhere.

Try it free
...

Write Tests in Plain English with KaneAI

Create, debug, and evolve tests using natural language.

Try for free

AI Governance Tools FAQs

Did you find this page helpful?

More Related Blogs

TestMu AI forEnterprise

Get access to solutions built on Enterprise
grade security, privacy, & compliance

  • Advanced access controls
  • Advanced data retention rules
  • Advanced Local Testing
  • Premium Support options
  • Early access to beta features
  • Private Slack Channel
  • Unlimited Manual Accessibility DevTools Tests