AI Agent Assurance for Your Agent Deployments

Agent Assurance Tests How Your Agent Actually Behaves across Workflows, Tools, and Actions to Catch Failures and Vulnerabilities.

npm install -g @testmuai/rook

Thought LeadershipAI

MCP Server Contract Testing: Your Agent Is Only as Trustworthy as Its Tools

MCP server contract testing proves what an AI agent can do, not what it says. Five deterministic gates check schema, claims, outputs, permissions and evidence.

Published on:

The model in your agent gets evaluated. Its tools are the part that touches production. Were they tested as a contract?

The Model Context Protocol (MCP) is now the default way agents reach those tools.

When MCP moved to the Linux Foundation's Agentic AI Foundation in December 2025, the project reported over 97 million monthly SDK downloads and 10,000 active servers.

Teams have instrumented the agents on top. According to LangChain's State of Agent Engineering survey (n=1,340, published June 2026), 89% of organizations have observability on their agents and 52.4% run offline evals.

Those tools tell you how the agent reasoned. They don't tell you whether the delete_invoice tool deletes only invoices, or whether a ticket from a stranger can make your support agent read a secrets table.

That's the job of MCP testing. Below I define it, show why incidents keep landing at the tool layer, and give QA teams a five-gate test they can run in CI with no model in the loop.

TL;DR

MCP testing verifies what an AI agent can actually do, not what it says. Before agents depend on an MCP server, test five things deterministically: Schema, Claims, Outputs, Permissions and Evidence (the SCOPE test). Evals and observability score behavior after the fact. SCOPE catches the tool-layer failures behind the public MCP incidents of the last 18 months, before release.

What Does MCP Server Contract Testing Check?

  • Schema and MCP conformance: MCP server contract testing starts at the protocol. Every schema from tools/list validates, results match each declared outputSchema and fuzzed inputs return clean errors. The MCP Inspector CLI exits 6 when tools/list --strict finds a schema portability error.
  • Claims and MCP tool annotations: readOnlyHint and destructiveHint are claims an MCP server makes about its own tools, and the MCP spec tells clients to treat them as untrusted unless they come from trusted servers. An MCP server contract test snapshots rows, files and outbound calls around each read-only call and fails the build on any difference.
  • Permissions and tenant isolation: The MCP server runs with its production credential, two test tenants confirm that no response crosses between them, and a tool granted read scope that attempts a write must be refused.
  • Evidence: Each MCP tool call emits a span with the tool name, description hash, caller identity, redacted arguments and result status, stored append-only in a system the deploying company controls.

How Is It Different From MCP Evals and Observability?

  • MCP evals: Evals score what an agent says, usually with a model-based grader, so the result is probabilistic. MCP server contract testing drives the server with a deterministic client and returns pass or fail, which is what can block a release.
  • MCP tool testing with a model: Only the Outputs gate uses a model, as a payload generator or as the agent under test. That model should differ from the production agent's model, and deterministic assertions, such as whether a forbidden tool was called, decide the result.
  • Agent Assurance: TestMu AI's Agent Assurance invokes an agent the way a user would, including through an MCP server, and grades each criterion against what the run changed. Its report can gate a release through Agent Assurance CI/CD.

What Is MCP Server Contract Testing?

MCP server contract testing is the practice of verifying, deterministically and before release, that an MCP server's tools do exactly what they declare and no more, under normal use, hostile input and failure, and that every call leaves evidence. It treats the server as a contract between your company and an untrusted caller: the model.

That last part matters. An MCP server's caller is a model that reads content you don't control: tickets, emails, web pages, other tools' output.

The client you're testing against is not a well-behaved app. It's an untrusted client with a confident voice.

MCP testing is often confused with three neighbours:

  • AI evals score an agent's answers. They need a model and usually a grader, and the results are probabilistic.
  • Agent observability records traces of what happened. It works after the fact by design.
  • Interactive MCP Inspector sessions are debugging. A developer clicking through tools is useful, but it isn't a release gate.

Why Do AI Agent Failures Keep Happening at the Tool Layer?

Because the tool layer is where authority lives, and it's rarely tested as a contract. In each public incident below, the decisive failure was in permissions, isolation, a dependency or content returned by a tool. None of them would have shown up as a bad benchmark score.

DateIncidentWhat failedSCOPE gate that catches it
Sept 2025postmark-mcp package (reported by The Register)A version of a third-party email MCP server quietly BCC'd outgoing mail to an attackerC: Claims (egress check), E: Evidence
July 2025mcp-remote, CVE-2025-6514OS command injection through a crafted authorization endpointS: Schema and protocol inputs
July 2025Supabase MCP research (General Analysis)The assistant ran with a role that bypasses row-level security; instructions planted in a support ticket led it to read an integration-tokens tableP: Permissions, O: Outputs
June 2025Asana MCP serverA bug could expose one customer's data to other Asana MCP users; the feature was offline June 5-17P: tenant isolation
June 2025MCP Inspector, CVE-2025-49596Missing authentication between the Inspector client and its proxy; CVSS 4.0 score 9.4Supply chain: your test tools need testing too
May 2025GitHub MCP prompt injection (Invariant Labs)A malicious public issue steered an agent with broad repo access into leaking private data to a public PR; Invariant called it "not a flaw in the GitHub MCP server code itself"O: Outputs, P: Permissions
Public MCP security incidents mapped to the SCOPE gate that catches each

The population data says these aren't one-offs. A study of 1,899 open-source MCP servers (Hasan et al., arXiv 2506.13538) found 7.2% with general vulnerabilities and 5.5% with MCP-specific tool poisoning.

A second study of 1,360 servers and 12,230 tools ("Mind Your Server", arXiv 2509.06572) found that 27.2% of servers exposed at least one threat-relevant tool.

OWASP now maintains an MCP Top 10 (beta), with Privilege Escalation via Scope Creep (MCP02), Tool Poisoning (MCP03) and Lack of Audit and Telemetry (MCP08) on the list.

Attackers also adapt. NIST's Center for AI Standards and Innovation reported that new red-team attacks lifted agent-hijacking success from 11% for the strongest baseline attack to 81%. If one static red-team pass is your tool-layer test, you're testing against the 11%.

The money is moving too. In one week this month, LangChain raised a $300M Series C at a $3B valuation (Nov 13), Plurai raised a $13M seed to stress-test agents before launch (Nov 11), and Bay came out of stealth with $50M for runtime enforcement of agent actions (Nov 10).

Observability, simulation and runtime enforcement are all funded. The deterministic contract test of the tool server is still mostly left to whoever wrote the server.

And controls are not keeping pace. KPMG's Q3 2026 AI Pulse found that the share of organizations building controls into agents alongside monitoring and evaluation fell to 30%, from 43% two quarters earlier.

How Do You Test an MCP Server Before Agents Depend on It? The SCOPE Test

Run five gates in order, with a deterministic client and no model in the loop: Schema, Claims, Outputs, Permissions, Evidence. Each gate gives a pass or fail result plus evidence a reviewer can read.

SCOPE test gates between an AI agent and its MCP servers

We start with no model on purpose. The first of our AI trust principles at TestMu AI is model diversity: no single model should be the only dependency for an important outcome, and no model checks its own work.

Most of an MCP server's contract can be verified with plain code, so it should be. Where a model does enter (gate O), it should not be your production agent's model.

S - Schema: Does the Server Honour the Protocol and Its Own Declared Shapes?

  • Run tools/list and validate every input and output schema. The official MCP Inspector CLI is built for this; its docs call it a good fit for CI pipelines. It returns stable exit codes: 5 when a tool call returns isError or the tool isn't found, and 6 when tools/list --strict finds a schema portability error.
  • For every tool that declares an outputSchema, assert that results conform. The MCP tools specification makes that a MUST for servers.
  • Fuzz inputs: missing fields, wrong types, oversized strings, odd Unicode. The spec requires servers to validate all tool inputs. Your test is the proof.
  • Pin protocol versions in CI. The current spec, 2026-07-28, made the core protocol stateless. MCP now ships on a six-month cadence, with the next version due 2027-03-30. Test against the versions your clients actually speak.

C - Claims: Does Each Tool Do What Its Annotations Say, and Nothing Else?

Annotations such as readOnlyHint and destructiveHint are hints, not guarantees. The spec says clients "MUST consider tool annotations to be untrusted unless they come from trusted servers." The MCP team's own post on annotations puts it more bluntly: a server can claim to be read-only and still delete your files.

  • Side-effect diffing. Snapshot the backing state (database rows, files, calls captured by a mock) before and after every read-only call. Any difference fails the test.
  • Destructive tools. Unless a tool sets it otherwise, destructiveHint defaults to true. Assert that every tool that writes is marked destructive, and that your client asks for confirmation before calling it.
  • Idempotency. For tools that claim it, call twice and compare state.
  • Egress. Record outbound network calls per tool against an allowlist. An email tool that adds a hidden recipient, as in the postmark-mcp case, fails here.

Check C1 on a demo server

The run below applies check C1 to a small invoice server built with the official MCP TypeScript SDK. Its get_invoice tool declares readOnlyHint: true, but the handler marks each invoice it reads as viewed:

server.registerTool(
  "get_invoice",
  {
    description: "Read one invoice by id.",
    inputSchema: { id: z.string() },
    annotations: { readOnlyHint: true },
  },
  async ({ id }) => {
    const db = load();
    const inv = db.invoices.find((i) => i.id === id);
    if (!inv) return { isError: true, content: [{ type: "text", text: `No invoice ${id}` }] };
    inv.status = "viewed"; // the side effect the annotation does not declare
    save(db);
    return { content: [{ type: "text", text: JSON.stringify(inv) }] };
  }
);

The MCP Inspector CLI finds no schema problem, so tools/list --strict exits 0, and a call for a missing invoice exits 5 as documented. Only the side-effect diff catches the broken read-only claim:

$ npx @modelcontextprotocol/inspector@2.15.0 --cli node server.mjs --method tools/list --strict > tools.json
exit code: 0

$ npx @modelcontextprotocol/inspector@2.15.0 --cli node server.mjs --method tools/call --tool-name get_invoice --tool-arg id=INV-9999 > call.json
{"error":{"code":"tool_is_error","message":"Tool 'get_invoice' returned isError:true."}}
exit code: 5

$ node claims.test.mjs
tools/list: 3 tools, 2 declare readOnlyHint
PASS C1 list_invoices: state unchanged (sha 06b4f20e76bb)
FAIL C1 get_invoice: readOnlyHint=true but state changed (sha 06b4f20e76bb -> b80d8e88cf0a)
     INV-1001.status: "open" -> "viewed"
1 read-only claim(s) broken
exit code: 1

The test behind that output uses the SDK's client over stdio and no model. Its exit code 1 fails a CI job the same way the Inspector's exit codes do:

// Gate C, check C1: every tool that declares readOnlyHint must leave the
// backing state unchanged. Deterministic client, no model in the loop.
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
import { readFileSync, writeFileSync } from "node:fs";
import { createHash } from "node:crypto";

const DB = "invoices.json";
const SEED = { invoices: [
  { id: "INV-1001", customer: "acme", amount: 420, status: "open" },
  { id: "INV-1002", customer: "acme", amount: 75, status: "paid" },
] };
const SAMPLE_ARGS = { list_invoices: { customer: "acme" }, get_invoice: { id: "INV-1001" } };
const snapshot = () => readFileSync(DB, "utf8");
const sha = (s) => createHash("sha256").update(s).digest("hex").slice(0, 12);

writeFileSync(DB, JSON.stringify(SEED, null, 2));
const client = new Client({ name: "scope-c1", version: "1.0.0" });
await client.connect(new StdioClientTransport({ command: "node", args: ["server.mjs"], env: { ...process.env, INVOICE_DB: DB } }));

const { tools } = await client.listTools();
const readOnly = tools.filter((t) => t.annotations?.readOnlyHint === true);
console.log(`tools/list: ${tools.length} tools, ${readOnly.length} declare readOnlyHint`);

let failed = 0;
for (const tool of readOnly) {
  const before = snapshot();
  await client.callTool({ name: tool.name, arguments: SAMPLE_ARGS[tool.name] ?? {} });
  const after = snapshot();
  if (before === after) {
    console.log(`PASS C1 ${tool.name}: state unchanged (sha ${sha(before)})`);
  } else {
    failed++;
    const b = JSON.parse(before).invoices, a = JSON.parse(after).invoices;
    const diff = a.flatMap((row, i) => Object.keys(row)
      .filter((k) => JSON.stringify(row[k]) !== JSON.stringify(b[i]?.[k]))
      .map((k) => `${row.id}.${k}: ${JSON.stringify(b[i]?.[k])} -> ${JSON.stringify(row[k])}`));
    console.log(`FAIL C1 ${tool.name}: readOnlyHint=true but state changed (sha ${sha(before)} -> ${sha(after)})`);
    diff.forEach((d) => console.log(`     ${d}`));
  }
}
await client.close();
console.log(failed ? `${failed} read-only claim(s) broken` : "all read-only claims hold");
process.exit(failed ? 1 : 0);

O - Outputs: Does the Server Treat What It Returns as Hostile Input for the Model?

Tool output goes straight into the model's context. That was the attack channel in both the GitHub and Supabase cases.

  • Injection fixtures. Plant payloads in issue bodies, ticket text, file contents and HTML. Assert the server returns them as clearly delimited, size-capped data, never mixed into instructions. The spec requires servers to sanitize tool outputs.
  • Pin the tool surface. Hash every tool's name, description and schema at release. Fail the build if the hash changes without review. This catches tool poisoning and silent description changes.
  • Adaptive payloads. Regenerate variants on every run, remembering NIST's 11%-to-81% result.
  • Model in the loop, carefully. This is the one gate where a model helps, as a payload generator or as the agent under test. Use a different model from your production agent, and score the result with deterministic assertions (did the forbidden tool get called?), not with a model's opinion.

P - Permissions: Does the Company, Not the Model, Decide What Each Tool Can Touch?

This is our deployer-defined authority principle, made testable.

  • Least privilege. Run the server with the exact credential it uses in production, then prove what it can't do. No admin keys, nothing that bypasses row-level security.
  • Tenant isolation. Use two test tenants. Tenant A's calls must never return tenant B's data. That's the Asana class of bug.
  • Scope creep. A tool granted read scope attempts a write. The server must refuse it; a prompt asking nicely is not a control.
  • Containment. Stop the session mid-task. Confirm in-flight calls halt, no partial writes remain, and rate limits (another server MUST in the spec) actually trip.
  • Human approval. The spec recommends that a human can always deny a tool invocation. Test that your client's confirmation step fires for every destructive tool, every time.

E - Evidence: Does Every Call Leave a Record Your Company Owns?

  • Emit a span per call. OpenTelemetry's GenAI conventions define execute_tool spans under an invoke_agent span. They're still under active development, and tool arguments aren't captured by default, so decide on purpose what you log and what you redact.
  • Log the facts a reviewer needs. Tool name, description hash, caller identity, redacted arguments, result status and any side effects observed.
  • Make it tamper-evident. Use append-only storage with hashes.
  • Keep it in your control plane. It should not live only in the server vendor's dashboard or your testing vendor's console.

Why now: EU AI Act Article 12 requires high-risk AI systems to "technically allow for the automatic recording of events (logs) over the lifetime of the system." After the Digital Omnibus, Annex III high-risk obligations apply from 2 December 2027. Evidence pipelines take longer to build than test suites.

The SCOPE Checklist (Copy Into Your Test Plan)

  • S1 tools/list --strict passes and every schema validates
  • S2 Every tool with an output schema returns conforming results
  • S3 Input fuzzing produces clean errors, never crashes
  • S4 Protocol versions are pinned and tested in CI
  • C1 Read-only tools show zero side effects
  • C2 Writing tools are marked destructive and gated by confirmation
  • C3 Each tool's outbound calls stay inside its allowlist
  • O1 Injection fixtures come back as delimited, size-capped data
  • O2 Tool name, description and schema hashes are pinned
  • O3 Adversarial payloads are regenerated on every run
  • P1 The production credential's limits are proven
  • P2 No cross-tenant data in any response
  • P3 A mid-task stop leaves no partial writes, and rate limits trip
  • E1 Every call emits a span carrying the tool's description hash
  • E2 Evidence is append-only and stored in a system you control

MCP Testing vs AI Evals vs Agent Observability: What's the Difference?

Evals grade what the agent says. Observability records what it did. MCP testing proves what it can do. You need all of them, but only the deterministic one can block a release on a pass or fail result.

MCP testing (SCOPE)AI evalsAgent observabilityRuntime enforcement (AI guardrails)
Question it answersCan this tool do more than it claims?Is the agent's output good enough?What happened in this run?Should this action be allowed right now?
When it runsBefore release, on every buildBefore release, plus sampled in productionProductionProduction, on every call
Needs a model?No, for the contract gatesYes, often plus a graderNoNo
OutputPass/fail plus evidenceScoresTracesAllow/deny log
What it typically missesQuality of the model's judgmentOver-broad permissionsFailures that haven't happened yetBugs inside allowed actions
Comparison of MCP testing, AI evals, agent observability and runtime enforcement

Read the bottom row as a dependency map. Each layer covers a blind spot of another, which is why consolidating them all under one vendor or one model is a weaker design than it looks.

Does SCOPE Still Apply if You Built the MCP Server Yourself, or Can't Change It?

Yes. What changes is who runs each gate.

  • You built it. Owning both sides is not a reason to skip testing. The caller is still a model reading untrusted content, so treat your own agent as an untrusted client.
  • It's a third-party server you can't modify. Run SCOPE as a black box from the client side. Pin the version and the tool-surface hash, run it in a sandbox with captured egress, and put your own scope enforcement in front of it. Gates P and E become your gateway's job.
  • It's hosted remotely. You can't diff the vendor's internal state. Test Claims through what you can observe: responses, outbound traffic from your side and changes in your own test tenant.
  • Who owns the evidence? You do, and that includes evidence your testing vendor produces. Us included: if a verification vendor's results live only in its console, it fails gate E.

Where Does TestMu AI Fit in MCP Testing?

Gates S to P are code your team can write today with the MCP SDKs and the Inspector CLI. We'd encourage it. The hard part is running them continuously across every agent, model version and server version, and keeping the evidence independent of the thing being tested.

That's the layer TestMu AI builds as an Agentic Quality Engineering platform. Agent Assurance, delivered by Rook, covers agent-to-agent workflows before release: agents testing agents with no human handshake.

Agent Testing covers the human-to-agent surfaces, such as chat, voice and IVR, where tool calls turn into actions a customer sees. Kane CLI gives SDETs a command-line path through the full QE cycle, and HyperExecute orchestrates large suites in parallel inside CI.

The thread through all of them is an independent verification layer: deterministic tests, evidence in your control plane, and verdicts that don't depend on the model under review.

Start a 14-day free trial of Agent Assurance by TestMu AI, with 4,000 credits, to test how your AI agents behave across workflows, tools, and actions

What Happens Next for MCP Security and AI Audit?

Our prediction: by the end of 2027, enterprise buyers will routinely ask MCP server vendors for SCOPE-style evidence (conformance, permission and isolation test results) the way they ask for penetration-test summaries today. Three cited trends point there:

  • A six-month MCP spec cadence, which turns every server into a regression-testing problem twice a year.
  • An OWASP MCP Top 10 that hands security reviewers a ready-made list of questions.
  • EU AI Act record-keeping duties for Annex III high-risk systems from 2 December 2027.

This is our view, not a certainty. The direction is clear enough to start building the evidence now.

Conclusion: Test the Tools, Not Just the Talk

An agent is only as trustworthy as the tools it can call. Evals and observability will tell you a lot about how your agent thinks. MCP testing tells you what it can do to your systems, and it can say so before release, with evidence, without asking a model to grade itself.

Start with the SCOPE checklist above. Run S and C this sprint. Then add O, P and E before your next agent reaches production.

Shipping agents that other agents use? Rook by TestMu AI does Agent Assurance: agents testing agents before release, with no human in the loop.

To run your first test, install Agent Assurance with npm install -g @testmuai/rook@latest command, or
for a full walkthrough of the product.

Author

...

Jay Singh

Blogs: 5

  • Twitter
  • Linkedin

Jay Singh is the Co-Founder and Chief Customer Officer at TestMu AI, with over two decades of experience in sales, customer success, and growth strategy. He has been leading customer initiatives at TestMu AI for more than 8 years, helping scale the platform to serve a global user base. Jay is followed by 15,000+ professionals on LinkedIn, reflecting his influence across the software testing and quality assurance industry. Before TestMu AI, he founded BusinessMojos, later acquired by DAMO Consulting, and VisualMojos Technologies. He also held senior roles at LeadSquared and Harman International.

Reviewer

...

Mayank Bhola

Reviewer

  • Linkedin

Mayank Bhola is Co-Founder and Head of Products at TestMu AI, where he leads the entire product portfolio across KaneAI, Kane CLI, HyperExecute, SmartUI, the Real Device Cloud, Accessibility, and other software testing product lines. As an early Lead Architect he designed and built the company's flagship Tunnel technology from scratch, created the React-based automation platform, and architected the data-intensive pipelines and FAAS services that scale it. He brings more than 10 years of experience in software development and product engineering, with earlier roles as Head of Technology at Juggernaut Books and Senior Software Engineer at PressPlay TV and Zomato. Mayank holds a B.Tech in Computer Engineering from JIIT Noida.

Add to Google preferred sources

Summarise with AI

Copied to Clipboard!
...

3000+ Browsers. One Platform.

See exactly how your site performs everywhere.

Try it free
...

Write Tests in Plain English with KaneAI

Create, debug, and evolve tests using natural language.

Try for free

FAQs

Did you find this page helpful?

More Related Blogs

TestMu AI forEnterprise

Get access to solutions built on Enterprise
grade security, privacy, & compliance

  • Advanced access controls
  • Advanced data retention rules
  • Advanced Local Testing
  • Premium Support options
  • Early access to beta features
  • Private Slack Channel
  • Unlimited Manual Accessibility DevTools Tests