What Is AI Trust?

What Is the Difference Between AI Trust and Trustworthy AI?

AI trust is an attitude held by people: the confidence that users, customers, regulators and the organization itself place in an AI system, and their willingness to rely on it when they cannot check every output. Trustworthy AI is a property of the system. Calibrated trust is the state in which the first tracks the second.

Reliance is the behavior that trust influences. In a 2004 paper in Human Factors, Lee and See argue that automation often causes problems because people fail to rely on it appropriately, and that trust guides reliance especially when complexity and unexpected situations make it impractical to understand the system completely.

NIST's AI Risk Management Framework lists the characteristics of trustworthy AI systems as valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed.

NIST calls valid and reliable "a necessary condition of trustworthiness" and the base for the other six. The framework's functions and testing guidance are covered in the NIST AI RMF guide.

A system can meet those characteristics and go unused, or miss them and still be relied on. Producing evidence that they hold for one specific system is the work of AI assurance. Keeping a system trustworthy, secure and compliant through policies and technical controls is the subject of AI TRiSM.

What Is Calibrated Trust in AI?

Calibrated trust is trust in an AI system that matches its verified performance, and it shows in reliance: people rely on the system for the tasks where evidence shows it works, and check or override it where the evidence is thin or poor.

Human factors research named the two ways to miss long before generative AI: misuse and disuse, as Parasuraman and Riley defined them in 1997.

Is More Trust in AI Always Better?

No. The aim is reliance that matches the evidence, and a research review and survey data both show people using AI more than they trust it, and the survey shows many relying on its output without checking it:

• Trust and adoption - a 2019 Partnership on AI review of 78 research articles found that, apart from some papers concerned about the dangers of overtrust, they largely share "the goal of fostering more trust in AI", and that "The link between trust and adoption is tenuous at best, as people often use technologies without trusting them."
• Use ahead of trust - in the University of Melbourne and KPMG global study of more than 48,000 people in 47 countries, run between November 2024 and mid-January 2025, 66% of people use AI regularly, while only 46% are willing to trust AI systems.
• Reliance ahead of checking - in the study's findings on AI at work, 66% rely on AI output without evaluating its accuracy.
• Mistakes at work - in the same findings, 56% are making mistakes in their work due to AI.

NIST makes a related point from the system side: trustworthiness "is a social concept that ranges across a spectrum and is only as strong as its weakest characteristics." So set reliance task by task: let people use an output unchecked only for tasks whose acceptance criteria have passed with evidence, and keep a check step everywhere else.

What Is the AI Trust Gap in 2026?

The AI trust gap is the difference between how much an organization trusts and relies on its AI systems and how much evidence it holds that they behave as intended. KPMG's Q3 2026 AI Pulse, which surveyed 314 US C-suite and business leaders at organizations with annual revenue of $1 billion or more between July 24 and August 25, 2026, shows the organizational side of it:

• Reliance on guardrails - KPMG's reason for the smaller share building controls into agents is that many organizations are counting on guardrails they already have, and the release does not say which kind. Runtime AI guardrails act while an agent runs, blocking or flagging inputs, outputs and tool calls that break a policy, so they limit what a live agent can do without showing how it behaves before release.
• The confidence control gap - Maneesh Sharma, TestMu AI's COO and the reviewer of this guide, uses that name for the pattern in his piece on AI agent governance. In this guide's reading, which goes beyond what KPMG reports, the pattern is over-trust at organizational scale.
• Trust against capability - in a Testμ Conf 2026 session, Moving from "Can We Build It?" to "Can We Trust It?", Andrew Duncan framed the trust gap as the space between the trust people place in an AI solution and what it can really do.

To size your own gap, list the actions each agent may take, mark which have evidence from tests run before release and which rest on runtime guardrails alone, and close the gaps for the actions people rely on most first.

How Do You Build Trust in AI Agents?

Build trust in an AI agent from evidence of its actions, gathered before it gets autonomy. A fluent reply proves nothing about the refund behind it: the refund agent's message can be accurate, or it can describe a refund that was never issued, was issued twice or went to the wrong order.

The README of Rook CLI, TestMu AI's command-line tool for testing agents, states the principle: "An agent's account of what it did is the weakest evidence available about what it did."

• Criteria written before any run - stakeholders agree in writing what the agent has to do, what it may never do and what it hands to a person, such as "never refund more than the order total". Criteria fixed before testing are the core of AI agent validation.
• Evidence of actions before release - each criterion is checked against what changed in staging: the refund record, the order status and the tool calls it made.
• Evals before deployment - a release decision rests on scored test results. In Build Trustworthy AI Agents Powered by Evals, a Testμ Conf 2026 session, Rushabh Mehta, Software Engineer at Meta, argued that vibes and a reviewer's "looks good to me" are not a deployment strategy.
• Gaps kept in view - criteria that end without a Pass or Fail, because nothing recorded could settle them, are reported with their own count and never added to the passes, so a reviewer sees what nobody checked.
• Review outside the build team - someone who did not build the agent reviews the evidence before the agent gets more autonomy.
• Autonomy granted in steps - the agent starts with actions a person approves one by one, and its limits rise only as the evidence for each action builds up.
• Re-checks after every change - a new model, prompt or tool reopens the criteria it touches, and trust carries over only once those checks pass again. In an agent development lifecycle, those re-checks run at a change gate.

Who Is Accountable When an AI Agent Is Trusted to Act?

The organization and the named people who release and run an AI agent should stay accountable for what it does, whatever share of the work, testing included, AI takes on.

• Liability stays with people - in a Testμ Conf 2026 fireside chat on who signs off when AI writes the tests, Harleen Bedi, Chief Technology Officer for Quality Engineering at Infosys, said: "End of it, liability still remains with humans. We as humans still own that." Her governance layer fixes criteria and thresholds before anything is handed to AI, traces why the AI wrote each test and marked it passed, and watches for drift after release.
• Decisions kept with people - in KPMG's Q3 2026 survey, 49% of leaders have defined high-risk use cases where they do not allow agents to make decisions autonomously.

For that accountability to hold, record each grant of autonomy: what the agent may do alone, the evidence behind the decision, who approved it and when, and which changes reopen it. When an outside reviewer later asks how the decision was made, that record is what an AI audit examines.

How Do You Measure AI Trust?

Measure AI trust from both sides and compare them: how much people rely on the system, and how well evidence shows it performs. Stated trust can be surveyed as well, but it often differs from what people do, so the comparison in this section uses logged reliance.

const agents = [
  // accepted / outputs: outputs staff used unchanged
  // passed, failed, unverified: criterion verdicts from
  // one pre-release test run
  { name: 'refunds', accepted: 455, outputs: 500,
    passed: 9, failed: 1, unverified: 14 },
  { name: 'order-status', accepted: 880, outputs: 1000,
    passed: 26, failed: 2, unverified: 2 },
  { name: 'address-change', accepted: 70, outputs: 200,
    passed: 17, failed: 1, unverified: 0 },
  { name: 'store-credit', accepted: 216, outputs: 300,
    passed: 8, failed: 4, unverified: 4 },
  { name: 'return-labels', accepted: 366, outputs: 600,
    passed: 7, failed: 1, unverified: 4 },
  { name: 'warranty', accepted: 30, outputs: 125,
    passed: 2, failed: 5, unverified: 3 },
];

const TOLERANCE = 10; // percentage points

function assess(a) {
  const all = a.passed + a.failed + a.unverified;
  const decided = a.passed + a.failed;
  const rely = a.accepted / a.outputs;
  const pass = a.passed / decided;
  const cover = decided / all;
  const shown = a.passed / all; // pass x cover
  const gap = Math.round((rely - shown) * 100);
  const call = gap > TOLERANCE ? 'over-trust'
    : gap < -TOLERANCE ? 'under-trust'
    : 'calibrated';
  return { ...a, rely, pass, cover, shown, gap, call };
}

Reliance vs evidence: illustrative numbers, not data

agent          rely  pass cover shown  gap  call
refunds         91%   90%   42%   38%  +54  over-trust
store-credit    72%   67%   75%   50%  +22  over-trust
warranty        24%   29%   70%   20%   +4  calibrated
return-labels   61%   88%   67%   58%   +3  calibrated
order-status    88%   93%   93%   87%   +1  calibrated
address-change  35%   94%  100%   94%  -59  under-trust

rely  = outputs used unchanged / all outputs
pass  = passed / decided criteria (Pass or Fail)
cover = decided criteria / all criteria
shown = passed / all criteria (pass x cover)
gap   = rely - shown, in percentage points
call  = over-trust above +10, under-trust below -10

How Do You Measure an AI Agent's Verified Performance With Agent Assurance?

Agent Assurance, the TestMu AI product built to test agents that act, supplies the verified-performance side of the comparison before release. You run it from the terminal as Rook CLI (command rook): it reads the agent's code or spec and generates functional and adversarial scenarios, each with acceptance criteria such as "the refund exists once, for the right order and amount".

Most eval and observability tools concentrate on the text an agent returns and the traces it leaves behind. Agent Assurance examines the changes a run made, within what your profile and access let it observe, and reports the unverifiable criteria on their own.

• Invoking the real agent - Rook CLI reaches the refund agent through a profile that names an MCP tool, a command or an HTTP endpoint; the profile's hook scripts, readable before any run, send each turn and can collect evidence afterward.
• Evidence of the refund - what the hooks report about tool calls and state changes is compared with what the scenario expects, including calls the agent must not make, and a judge can also check the refund through a read-only tool, once you have approved that tool.
• Read-only by instruction - judges are told to check without writing anything: the agent's refund tool issues refunds, so a judge that called it to check would create another one. Give the verifying tool read-only credentials as well.
• What the reply says - the reply is scored on quotations of what the agent actually wrote, a model judge decides some criteria, and the reply never counts as proof that a refund happened.
• Criteria left open - each criterion is marked Pass or Fail, or else Unable to Verify if no evidence settles it. Unable to Verify gets its own count and is never folded into Pass or Fail, and the pass rate is a scenario count: scenarios that passed, out of those that reached a decided verdict.
• Saved with each run - a record a reviewer can reopen later: each scenario as it stood for that run, which agent version ran under which profile revision, and every verdict beside the quote it rests on, kept in your project as plain files.
• Run-to-run changes - after a second run, Rook CLI compares it with the previous one, showing how the scenario pass rate moved and naming the scenarios that began failing or got fixed. Because an agent may choose another route through the same input, a sign-off rests on these records instead of a promise that the next run will match.
• Where to point it - a staging copy of the payment system with synthetic orders, because a refund the agent issues during a run is a real write that Rook CLI cannot take back. These checks belong before release: Agent Assurance does not see the refunds a live agent issues and cannot halt one at runtime.

Rook CLI works on macOS and Linux, and 64-bit Windows through npm or WSL. To set it up from npm, and to add its skill if Claude Code drives your work, run:

npm install -g @testmuai/rook@latest
npx @testmuai/rook-skill@latest install \
  --agent claude-code

• npm - requires Node.js 20 or newer on the machine.
• Homebrew or the shell installer - works without Node, and the Rook CLI installation guide gives the command for each.
• Claude Code skill - set up by an npx installer that runs on Node.js 22 or newer.
• Plan or trial - Agent Assurance is generally available, its free trial runs for 14 days with 4,000 credits, and the self-serve plans are compared on the Agent Assurance plans page. Rook CLI will not call a model or spend credits until your TestMu AI account has an active plan or trial.

To confirm a refund without asking the agent, follow the guide to configure MCP servers in Agent Assurance, which shows how a judge calls an approved read-only tool.

Where Should You Start Calibrating AI Trust?

Start with the agent your team relies on most: pull a month of its outputs and count how many were used unchanged, edited, overridden or redone. Then write its acceptance criteria, test them against staging, and set the reliance rate beside the criterion pass rate and coverage before you widen or narrow what the agent may do alone.

For an agent that acts in other systems, the Agent Assurance quickstart covers a first run on a public sample agent. If the agent talks to people or analyzes images, use Agent Testing instead, a separate TestMu AI product that puts chat, voice, phone and video agents through simulated conversations and scores image agents on test images, before launch and after model updates.