evidence-cli: An Open Format for Test Run Evidence

One Shape, Whatever Made It

A browser agent, a Playwright suite, a Jest run, an API check. Each can emit the same pack, and a reader can open any of them without knowing the producer. The container is open so anything can write one, and any tool can read one.

This matters most where a green tick is a claim rather than proof. A passing report that cannot be inspected asks you to trust it. A pack holds the artifacts behind the verdict, so the tick has something underneath it.

Anatomy of a Pack

A .evidence pack is a directory, or a sealed zip, built around three required files:

• run.yaml - the manifest anchor. It describes the run itself: who produced it, when, and against what.
• The definition - the test as it was written, opaque by design. The format references it and hashes it, and never parses it. It can be a test.md from Kane CLI, a login.spec.ts from Playwright, or anything else.
• result.yaml - the verdict and the step-level outcomes.

Around those sit the artifacts: screenshots, network captures, logs, whatever the producer chose to include. Keeping the definition opaque is what lets one container hold a YAML flow and a TypeScript spec without the format needing to understand either. The replayable test format that Kane CLI writes is covered separately in Test.md.

Profiles, Not Rewrites

The format scales through a profile ladder rather than through breaking changes. L0 is the minimal, framework-neutral core. L1 is purely additive: it keeps all of L0 and adds requirements for the captured evidence artifacts on top, never rewriting what L0 means.

The contract version is a separate axis. It defines the meaning of the fields, and only a breaking change to an existing meaning bumps it. Adding a profile is additive, so the ladder grows without a migration.

The two CLIs default differently, which is the detail most likely to surprise you. Reading the constants out of the installed package settles it:

$ grep -E "DEFAULT_PROFILE|CONTRACT_VERSION" dist/contract.js
exports.CONTRACT_VERSION = "0.1";
exports.DEFAULT_PROFILE = "L0";

So the standalone evidence CLI validates against L0 unless told otherwise, while kane-cli evidence validate declares L1 as the default on its own --profile flag. Same format, stricter bar inside the product that produces the packs.

Inside the Published Package

Method for the table below: install the package from npm into a clean project, read the shipped files, and run the documented entry points. Every value is what the artifact reports, not what the docs claim.

The named exports resolve cleanly from an ES module project, which is worth confirming rather than assuming for a package whose build output is CommonJS:

import { validate, finalize } from "@testmuai/evidence-cli";
console.log("validate:", typeof validate, "finalize:", typeof finalize);

// validate: function finalize: function

validate is async and takes a target plus options, so a pack check is a single awaited call against a directory or a sealed zip. Validation exits 0 for a valid pack, 1 for an invalid one, and 2 for a usage error, which means a pipeline fails on it without extra scripting.

Capture Is the Producer's Job

The tooling captures nothing. It drives no browser and executes no tests. It validates a pack, derives what can be derived, and seals it. A screenshot is in your pack because the producer put it there.

That split is deliberate, and it is what makes the container open. Kane CLI is the reference producer: every run seals a pack into .testmuai/evidence/ holding results, per-step screenshots with annotated copies, a HAR network log, console output, and the failure record. A pack is a file you keep and diff in git, not a third-party artifact on a retention timer. If you are weighing CLI options for that job, AI browser testing CLIs compares the field.

If your framework already produces results, it can produce a pack:

# As a library (Node 18+)
npm install @testmuai/evidence-cli

# Or the CLI, globally
npm install -g @testmuai/evidence-cli

Open Governance and Getting Started

The repo is its own spec. Every change starts as a decision record covering the proposition, the options, the decision, and the reasoning, and the project states the rule plainly: no code lands without a decision, and no change lands without updating the structure. The JSON Schemas are the single source of truth, consumed by both the validator and the viewer.

Start by installing the package and validating a pack you already have, then point your own framework at the worked example and emit the same shape. The spec, the contract, the changelog, and the design log all live in the repo, and the Kane CLI documentation covers the producer side if you want packs without writing an emitter first.