AI Agent Assurance for Your Agent Deployments
Agent Assurance Tests How Your Agent Actually Behaves across Workflows, Tools, and Actions to Catch Failures and Vulnerabilities.
npm install -g @testmuai/rook
Benchmark Contamination: Why Your Agent Evals Need a Sealed Suite
Benchmark contamination turns AI eval scores into recall tests. Learn how leaks happen, how to detect them, and the Sealed Suite framework for agent evals.
Published on:
The first ten days of November saw the people who build AI benchmarks go out and do a whole lot of auditing and fixing. Prime Intellect audited three public evals and found graders rejecting valid answers and agents pulling solutions off the open web.
A paper submitted on the 10th of November found that on SWE-bench Verified, the top 10 agents next to each other differ by less than the errors in the benchmark itself.
Google and SemiAnalysis both released agent benchmarks that hold tasks back from public view.
Mercor raised $750M at a $20B valuation and builds custom evals based on the actual workflows of each enterprise.
The signal is consistent. Public scores are getting weaker, as release evidence and benchmark contamination are a big part of that. If the teams that run the frontier benchmarks are locking their task sets, then the eval suite that gates your agent release should be locked too.
This is for the QA leads and SDETs who have that gate. It's about how contamination gets into an internal suite, how you spot it, and four controls we refer to as the Sealed Suite.
TL;DR
Benchmark contamination occurs when the test cases (or their answers) get to a model or agent prior to the test running via training, tuning, prompts or retrieval. The score then becomes a measure of recall rather than capability. The solution for teams who are shipping agents is a Sealed Suite: private test cases, independent authors and graders, deterministic end state checks and a hashed record of every gated release.
Contamination Routes in AI Evals and LLM Evaluation
- Public benchmarks: In public LLM evaluation, benchmark items can leak into training data, so a high leaderboard score may reflect recall. Public results help shortlist models, but they do not test your agent's policies, tools or data.
- Internal eval suites: Internal AI evals get contaminated with no training run at all: engineers tune prompts against the release cases, paste hard cases into context, or let the agent's own model family write and grade the tests.
- Eval harness access: An eval harness that stores sealed cases where the agent's search or API tools can reach them turns a test into a lookup. Keep the holdout readable only by the release pipeline and a named eval owner.
Sealed Suite Evidence for Assurance and Governance
- AI assurance: Assurance needs evidence a reviewer can check without trusting the team that built the agent. A Sealed Suite's evidence record logs the suite hash, agent build, model, verdict and evidence for every gated run.
- AI verifiability: Verdicts are verifiable when grading starts with end-state assertions, such as a database row written or a tool called with the correct arguments, because the same case, build and end state give the same verdict.
- AI governance: An AI governance framework can cite a hashed, versioned holdout as test-data evidence, mapped to NIST AI 600-1 action MS-2.11-001 and to the Article 10 data duties for high-risk systems under the EU AI Act.
- Agent Assurance: TestMu AI's Agent Assurance runs generated scenarios against the real agent before release, checks what the run changed rather than what the agent reported, and reports separately what it could not verify.
What Is Benchmark Contamination?
Benchmark contamination is any route by which a test case (or its expected answer) gets into the system under test prior to the test actually running. Once that has happened, a pass just tells you that the system has seen the answer; it doesn't mean that the system can produce the answer.
There are four paths, and only the first one is about training:
- Training data. The test items end up in the pre-training or fine-tuning data. The GSM1k study essentially recreated a grade school maths benchmark from scratch and saw accuracy drops of up to 8%, with some model families exhibiting clear signs of systematic overfitting and a number of the frontier models showing very little.
- Tuning. The engineers are iterating prompts, tools, and routing against the same cases until the cases pass. Nothing has leaked into the weights; it's just been tuned to the suite.
- Context: Eval cases get pasted into few-shot examples, system prompts, RAG indexes or agent memory.
- Retrieval at runtime. An agent with search or API tools finds the answer. Prime Intellect has documented models with web access retrieving solutions from Hugging Face and one model finding the graded rollout of another model via a public API and submitting its graded answer as its own.
For an enterprise team, paths 2 to 4 are the most relevant. You won't be training the foundation model. You'll be tuning against your own suite every day, and your agent will have tools.
Why Are Public AI Benchmarks Failing as Evidence?
Public benchmarks fail as release evidence in four ways: they leak, they have errors in them, they cease to separate the systems, and they go stale. Any one of those can turn a score into noise. Your internal suite can fail in the same four ways.
| Failure | What you see | Evidence | What to do in your suite |
|---|---|---|---|
| Leaked | Scores rise without a capability change | GSM1k: accuracy drops of up to 8% on fresh items (arXiv 2405.00332) | Keep a holdout the tuning loop never sees |
| Wrong | The expected answer is wrong, or the grader rejects valid answers | An estimated 6.49% of MMLU questions contain errors, and 57% of the analysed Virology questions do (arXiv 2406.04127). Reviewing SWE-bench led OpenAI to filter out 68.3% of samples as underspecified or unfairly tested (OpenAI) | Expected outputs need human review and an owner |
| Saturated | Top systems tie within measurement error | Neighbouring top-ten agents on SWE-bench Verified differ by less than the benchmark's own test errors (arXiv 2611.13541) | Retire cases everything passes; add cases from new failures |
| Stale | References drift from reality | SemiAnalysis found older GDPval tasks with broken links and outdated prices (GDPx) | Date every case and re-check references on a schedule |
The builders' response is very telling. The Google Multimodal Knowledge Agents benchmark keeps its 200 tasks private so they don't leak into the training data or get found via search and publishes a separate set of 10 public tasks.
ExpertCrowd-SWE publishes 24 of its 100 tasks and holds back 76. The pattern is the same: a small public set to show the format and a sealed set for the actual scoring.
How Does Contamination Get Into an Internal Eval Suite?
Internal contamination can come entirely from the team's own workflow: the tuning loop, the copied context, and the models that write and grade their own tests. Nothing malicious about it. It's just what happens when your dev loop is also your release gate.
Six routes we look for:
- One set, two jobs. The cases the engineers debug against every day are the very same ones that determine whether the gate is opened or not. Every fix makes the gate easier to go through.
- Copied context. A hard case gets pasted into the system prompt as a few-shot example "just to fix it".
- Generator overlap. The synthetic cases are generated by the same model (or model family) that runs the agent. Research into LLM evaluators has shown that models can identify their own outputs and will tend to favor them. So, same-family generation and grading will bias the result.
- Grader overlap. The agent's model is grading the agent's own answers. That violates TestMu AI's very first AI trust principle of model diversity (no model should be grading its own work).
- Reachable answers. The cases are in a public repo, a wiki page or a ticket that the agent can search.
- Unowned references. The answers that were correct when written were wrong six months later when the policy/price/API had changed, and nobody had updated them.
The exposure is wide. In the LangChain State of Agent Engineering survey of 1,340 practitioners, 52.4% of organizations were doing offline assessment on test sets and 32% cited quality as a top barrier.
For a lot of those teams, the test set is the one that ultimately decides whether an agent gets released, so it's very much the same thing as the agent itself in terms of the level of scrutiny it warrants.
The Sealed Suite: A Framework for Contamination-Resistant Agent Evals
A Sealed Suite is an agent eval suite where the cases, the answers and the grading logic are all completely out of reach of the system under test and the people who are tuning it. There's a tamper-evident record of every run. It has four seals, and each of those seals can be tested so an auditor can go and check it without having to trust the team that built the agent.
Seal 1: Sealed Cases (Private Holdout)
- Split the suite into a dev set that the builders can use every day and a sealed holdout that the builders can't see at all. Only the release pipeline and a named eval owner (who doesn't get to tune the agent) have access. Reviewers see a case once, at promotion.
- Put a unique canary string in every sealed case file. If the canary turns up in a prompt repo, a RAG index, a fine-tuning set or a ticket, then that case is burned.
- Rotate: retire the burned cases and the cases every candidate passes and replace them with cases from the recent production failures.
- Don't publish anything from the holdout. If you want to show the format, then publish the individual examples as Google's MKA benchmark does with its 10 public tasks.
Seal 3: Sealed Grading (Deterministic First)
- Assert on the end state wherever you can: the database row that's been written, the tool that's been called with the correct arguments, the API side effect, the refund that hasn't been issued. Those are the ones that are deterministic and repeatable.
- Use the rubric or LLM judges only where you can't assert something. Take them from a different model family to the agent and calibrate them against human labels.
- Add control answers: known-lazy and known-wrong outputs the grader must fail. ExpertCrowd-SWE does this with control patches in nearly half of its tasks. A control that passes means the gate is broken, not the agent.
- Run each sealed case more than once. In τ-bench, gpt-4o's pass^8 was below 25% in the retail domain, so a single pass tells you very little about reliability.
Seal 4: Sealed Record (Evidence)
- Version and hash each holdout release. For every gated run, log the suite hash, agent build, model and version, the verdict and the evidence for the verdict.
- Keep that record in the deployer's own control plane (your CI artifacts, your storage, your retention policy). Evidence that a vendor has for you is evidence that you can lose.
- Map it to what auditors already ask for. The NIST Generative AI Profile suggests verifying TEVV data provenance (MS-2.5-005) and documenting any possible training/test data cross-contamination in the benchmarks (MS-2.11-001). This is all voluntary guidance. For high-risk systems under the EU AI Act, Article 10(3) requires that the test data sets be relevant, sufficiently representative and, to the best extent possible, free of errors.
Sealed Suite Scorecard
Score one point per 'yes'. Items 1, 6, 10 and 12 are gates (one gate per seal). If you've missed any gate or scored below 9 of 12, treat your eval scores as indicative only, not as release evidence.
- Dev set and sealed holdout are separate, and the builders can't see the holdout.
- Every sealed case has a canary string, and we look for that on a monthly basis.
- There's no sealed case in the prompts, the few-shot examples, the RAG indexes or the fine-tuning data.
- The agent's tools can't get to the sealed cases.
- At least some of them are drawn from actual production incidents.
- Synthetic cases come from a different model family and are human-reviewed prior to promotion.
- Cases cover authority boundaries: refusals, forbidden data, required escalations.
- The vast majority of the verdicts are based on deterministic assertions about the end state.
- Any LLM judge comes from a different family to the agent and is judged against humans.
- Control answers do exist, and the grader will fail them.
- Each sealed case runs several times per release, and you report pass^k, not just pass@1.
- Suite versions are hashed, and every release record names the suite hash that it passed.
How Do You Detect Benchmark Contamination in Your Own Evaluations?
Watch the gap between your dev set and your sealed holdout. If your dev scores go up and your holdout scores don't, then you're fitting to the suite. That's exactly the logic that GSM1k applied to the public maths benchmarks, just applied to your own situation.
Five more leak tests:
- Canary scan. Search the prompt repos, vector stores, fine-tuning data, and ticketing systems for canary strings.
- Paraphrase test. Re-word a sample of sealed cases without altering the intent. A large drop on the paraphrases is a clear indicator of memorized surface patterns.
- Trace audit. Go through the agent traces to look for any tool calls that would have an effect on where the eval cases or answers are stored.
- Completion probe. On models you fine-tune, feed in the first half of a sealed case and see if the model completes the correct answer. Take that as a signal, not as proof, and drop every case you probe on: the act of probing reveals it.
- Fresh case check. On each cycle, add a batch of new cases taken from incidents. If the agent performs significantly worse on the new cases than on the old ones, then the old ones have either leaked or saturated.
What About In-House Models, Single Vendor Shops and Small Teams?
The seals still apply; only the mechanism changes. Four edge cases come up often:
- One approved model family. If procurement will only allow one family, then don't let it grade itself. Lean very heavily on the deterministic assertions and use human labels for authorship and for anything that still needs judging.
- You fine-tune. The training data is deduped against the sealed cases by hash prior to every training run, and the holdout is kept in a store that the training pipeline can't read.
- Very small team. A small sealed holdout beats none. Langfuse's point that coverage of distinct failure modes is more important than raw count certainly holds. Our golden dataset guide covers sizing and versioning.
- A vendor runs your suite (including us). The suite and its evidence should reside in your repository and your CI. A verification vendor should run the gate and return the evidence, not become the sole home for that evidence. Ask any vendor (us included) two questions: which model family generates the scenarios and grades them, and can you export every transcript, recording and report into your own storage?
Where Does TestMu AI Fit in a Sealed Suite?
TestMu AI provides the independent verification layer that runs the sealed cases and feeds the evidence back into your pipeline. It's a completely separate system from the agent that it's checking, which is the whole point.
- For human-to-agent interactions (chat, voice, IVR), Agent Testing can auto-generate 60 to 100+ scenarios from PRDs as well as from Jira and Confluence. In a Sealed Suite, those are the Seal 2 candidates that go through human review. Each session returns a recording and a turn-labeled transcript with timestamped evidence. Continuous runs in CI diff each run against the last.
- For agent-to-agent workflows, Agent Assurance (from Rook) generates functional and antagonistic scenarios, checks the impact on tools, files and APIs and reports what it couldn't verify. Each verdict is backed up by its evidence. The JSON report comes with gate recipes for GitHub Actions, Jenkins and Argo CD so the record gets into your control plane.
- When the agent is driving a web or mobile app, Kane CLI will save the steps, the screenshots, the console and network logs and the verdict into an .evidence file for each run. HyperExecute will then run the suite at scale in CI.
The checks that determine a verdict must be deterministic. Given the same sealed case, the same build and the same end state, you should get the same verdict.
The agent under test may well be non-deterministic. The gate itself should not be.
Sealed Suite Maturity Model
| Level | Stage | What it looks like | Gap |
|---|---|---|---|
| 0 | Vibe check | Someone tried it and it seemed fine | No repeatable cases |
| 1 | Leaderboard | The model vendor's public benchmark scores | Measures the model, not your agent or your policies |
| 2 | Shared regression set | One internal set used for both dev and release | The tuning loop contaminates the gate |
| 3 | Sealed holdout | A private holdout only CI can read | Authorship and grading may still share the agent's model |
| 4 | Sealed Suite | All four seals, plus a hashed evidence record | Needs ongoing rotation and an owner |
What Happens Next?
We're expecting three shifts. Those are our forecasts, each one linked to a trend that is already manifest.
- By the end of 2027, enterprise AI buyers will be asking for results on buyer-held private suites rather than leaderboard scores. Mercor co-CEO Adarsh Hiremath, quoted by SiliconANGLE, said that the models can pass the public benchmarks but will fail on 'the systems, policies and edge cases that define how a specific business runs'. The public benchmark builders are going private as well.
- Eval-set provenance will be an audit artifact. The EU's Digital Omnibus (Regulation (EU) 2026/1744) has moved the Annex III high-risk obligations to 2 December 2027. That gives teams just over a year to demonstrate that their test data satisfies Article 10, and a hashed, versioned holdout is the easiest way to do that.
- Evaluation is increasingly getting folded into the security and observability platforms. CrowdStrike agreed to acquire Fiddler AI on 3 November, with the acquisition being based on pre-deployment evaluation and drift detection. Consolidation is of course useful, but it makes Seal 4 even more important. Whoever is running the tools, the sealed cases and the evidence need to remain with the deployer.
Conclusion
Benchmark contamination isn't just a problem in the frontier labs. It's a problem anywhere you're using the same cases for tuning and for release decisions and where the agent can see the answers. The teams that are putting the public benchmarks together are in the process of finalizing their task sets.
Do the same for the suite that gates your agent: private cases, independent authors and graders, deterministic end state checks and a hashed record that you own. A score is only evidence if the test was sealed.
Start with the scorecard above. Score the suite that gates your next release against the 12 items, fix any missed gate first (items 1, 6, 10 and 12), and split a sealed holdout out of your regression set before that release ships.
Shipping agents that other agents use? Rook by TestMu AI does Agent Assurance: agents testing agents before release, with no human in the loop.
npm install -g @testmuai/rook@latest command, or for a full walkthrough of the product.
Author
Jay Singh
Blogs: 5
Jay Singh is the Co-Founder and Chief Customer Officer at TestMu AI, with over two decades of experience in sales, customer success, and growth strategy. He has been leading customer initiatives at TestMu AI for more than 8 years, helping scale the platform to serve a global user base. Jay is followed by 15,000+ professionals on LinkedIn, reflecting his influence across the software testing and quality assurance industry. Before TestMu AI, he founded BusinessMojos, later acquired by DAMO Consulting, and VisualMojos Technologies. He also held senior roles at LeadSquared and Harman International.
Reviewer
Maneesh Sharma
Reviewer
Maneesh Sharma is Chief Operating Officer at TestMu AI, where he leads the company's revenue functions across go-to-market, sales, RevOps, customer success, and the partner ecosystem, driving its global expansion. He brings over 28 years of experience across product, go-to-market, and business operations. Before TestMu AI he was Managing Director and General Manager for GitHub across India and APAC, Head of Partner Sales and Ecosystem at Adobe, Chief Revenue Officer at WizIQ, and Head of Products and Solutions at SAP India. Maneesh holds an MBA from IIM Bangalore and a B.E. in Electronics and Communications from NIT Surat.
FAQs
Did you find this page helpful?
More Related Blogs
TestMu AI forEnterprise
Get access to solutions built on Enterprise
grade security, privacy, & compliance
- Advanced access controls
- Advanced data retention rules
- Advanced Local Testing
- Premium Support options
- Early access to beta features
- Private Slack Channel
- Unlimited Manual Accessibility DevTools Tests