Next-Gen App & Browser Testing Cloud

Trusted by 2 Mn+ QAs & Devs to accelerate their release cycles

How Do We Prove to Auditors What an Automated Test Did?

By Anupam Pal Singh. Last reviewed November 14, 2026.

You prove what an automated test did with the record the run left behind, not its pass or fail line: which test version ran, who or what started it, where it ran, each step it took, and what it saw. Keep that record (video, step screenshots, logs or a sealed evidence pack) for the full audit period.

The KaneAI test run instance view docs show one such record: an Evidence Reporting run is sealed into a pack with the steps, screenshots, logs, verdict and AI failure analysis for each test instance, Autoplay walks through its steps in order, and the downloaded .evidence file opens for anyone who has it. The view is rolling out in phases, for test cases whose code was generated after it was enabled.

What Does an Auditor Need From an Automated Test Run?

Before you cite a run as evidence, check that its record answers these questions:

Which test ran: the definition or test case version that executed.
Who started it, and when: a user or a scheduler, with a timestamp.
Where it ran: browser, OS, resolution, device and framework.
What it did and saw: each step in order, with its screenshot, video and logs.
How the verdict was reached: assertions, retries, any step that auto-heal recovered, and any muted test left out of the build verdict.
What changed later: edits to the test, its results, remarks or assignee.

Which Artifacts Does One Test Run Leave Behind on TestMu AI?

Per the docs, one run can leave this record on each product:

Product	What one run records	Documented in
Web and app automation	Session video (on by default), command logs and Selenium or Appium logs; screenshots at each Selenium command, console logs and network (HAR) logs once their capabilities are on	Debugging options
HyperExecute	Job artifacts you declare under `uploadArtifacts` in the YAML, such as reports and logs, downloadable from the dashboard	Job artifacts
KaneAI	Status, duration, configuration, test case version, who or what ran it, each step, screenshots, video and timestamped logs	Test run instance view
Kane CLI	One sealed .evidence pack with the test definition, result summary, per-step screenshots, logs and failure records	Kane CLI evidence packs
Test Manager	Test case versions and an execution log of status, remark, attachment and assignee changes	Test instance audit logs
Agent Assurance (Rook CLI)	Request, response, observed calls, and a verdict with evidence for each acceptance criterion	Agent Assurance results and evidence

Set the visual, console and network capabilities to true for any suite an auditor may review. Per Selenium automation capabilities, video records up to 10 minutes and step screenshots stop at 150.

What Does an Evidence Replay of a Single Run Look Like?

To replay one Kane CLI run for an auditor, use these steps, built from the Kane CLI evidence docs:

Validate the pack: kane-cli evidence validate <execution-id-or-path> confirms it is intact and complete, running the full seal checks only on a finalized pack. A test marked passed that contains a failed step gets a warning, not a failure (see validating evidence packs).
Open it: kane-cli evidence serve .testmuai/evidence/<execution_id>.evidence starts a local server on 127.0.0.1 and prints a viewer link; the viewer reads the pack from your machine, and nothing is uploaded (see viewing evidence packs).
Show what was asked: the test definition, then the result summary with status, duration, per-step outcomes and who ran it.
Step through it: each step's screenshot, an annotated copy marking the element acted on, and the console and network logs that step produced.
Explain any failure: a failed step's record gives the error message and the page state at failure.

The open .evidence format lets an auditor read a pack without knowing which framework wrote it. KaneAI's Evidence Report gives a similar step-by-step replay in the web app when every test case uses New Experience and every instance runs on Chrome desktop or a real-device mobile app (see KaneAI test runs on HyperExecute).

How Long Is Test Evidence Kept, and How Do We Keep It for an Audit Period?

The data retention policy sets these defaults by product:

Data	Default retention
Web and app automation videos, logs and screenshots	30 days
HyperExecute job artifacts and reports	30 days
KaneAI sessions, including generated code, runs and version history	No retention limit
Test Manager projects, test cases and test runs	No retention limit while the account is active

Automation data linked to a Test Manager test case still follows the automation product's policy, and expired data cannot be recovered. The policy gives KaneAI sessions no limit but HyperExecute artifacts 30 days, so for KaneAI tests run on HyperExecute, confirm with support which period applies. For a longer audit period, Enterprise customers can request extended retention for most data types; otherwise export logs through the REST API, download HyperExecute artifacts and .evidence packs, and store them with your audit records. Session videos and screenshots follow the same 30-day default, so save the ones an audit needs before they expire.

Where Does TestMu AI Fit, and What Can Test Evidence Not Prove?

On TestMu AI, the record comes from the product that runs the test: video, screenshots and logs on the test automation platform and HyperExecute, evidence packs from KaneAI and Kane CLI, versions and execution logs in Test Manager, and verdicts with evidence from Agent Assurance (Rook CLI) for AI agents that act. The evidence has three limits:

Authorship: packs are not signed, so the recorded hash shows the definition is unchanged, not who produced the pack (see the .evidence format docs).
Coverage: a pass rate does not show coverage. An example in the Agent Assurance docs pairs a 100% scenario pass rate with 40% criterion verification coverage.
Fit to the control: the record proves what ran; mapping each test to the control it covers belongs in a traceability matrix.

Frequently Asked Questions

Can an auditor without a TestMu AI account review a test run?

Yes, for KaneAI runs in the new test run instance view, which is rolling out in phases: its share link works without a TestMu AI account until the expiry you set, such as 7 days. On the automation cloud, the Share dialog sets an expiry and emails the link, and the sharing test results guide notes you must be logged into TestMu AI to access the sharing URL, so confirm an outside auditor can open it first.

Can someone change a test after it ran and hide what was tested?

In Test Manager, not by editing: every saved edit creates a new version, as test case versioning describes, and a KaneAI run records the version it used. Deleting a test case destroys its version history, past results and audit logs, so archive test cases instead (see archiving and restoring test cases). On the automation cloud, the test detail page lets a user rename or delete a test, so export the logs you rely on.

How do we keep passwords and card numbers out of test evidence?

Mask them where you can. On the automation cloud, the maskCommands capability replaces typed text and cookie values with asterisks in session logs, and the network.mask capability masks values of keys such as password, token and auth in network logs. Kane CLI masks variables marked secret: true in displayed output and logs. These settings cover logs and displayed output; no doc says screenshots, video or an evidence pack are masked, so test with dummy accounts and card numbers.

Does using a certified testing platform make our tests compliant?

No. The TestMu AI security page lists SOC 2 Type II, ISO 27001:2022, ISO 27017:2015 and ISO 27701:2019 certifications, which cover TestMu AI's own controls. Whether your test evidence satisfies a regulation or a control is for your compliance team and auditor to decide.

What does Unable to Verify mean in an Agent Assurance report?

It means the available evidence was not enough to establish the outcome, which is not automatically an agent defect. The docs define the pass rate over decided verdicts only, so report Unable to Verify results to an auditor as a separate count.