Proving Agentic Test Runs With .evidence [Testμ 2026]

Fair To Today’s Reports

He opens on what every team produces constantly and then throws away. Not whether the test passed, but what the run produced: the screenshot, the network, the console, the thing the browser saw at the moment it went wrong. All of it gets created and almost none of it survives.

He sets out the structure explicitly and keeps to it: be fair to what exists today, give four numbered problems, then show what was built and what was opened up. He does return at the end and close all four in order.

For anyone who missed earlier sessions, he recaps Kane CLI in a sentence: describe what you want to test in plain English and it drives a browser or a mobile app from your terminal, from CI, or agent-driven. That is a recap of prior sessions, not something that happens here.

The reason Kane CLI matters to this talk is not the obvious one. It is the first producer of the format he is about to show, and he stresses first rather than only, then parks it until the end.

His status-quo slide is a screenshot of an existing test report showing nearly 5,000 tests and almost 99% green. He pre-empts the objection: he is about to spend ten minutes on what is missing and does not want it heard as a drag, because this is a good report. The tool that produced it is not identifiable from the recording and is not named here.

He itemises what it does well: a trend line across the last twenty-odd builds, the environment, the CI job that started it, a breakdown by feature, and somebody having gone to the trouble of splitting product defects from test defects, which he calls a thoughtful thing. His question is narrower. What can it not tell you, even in principle?

Problem One: Trapped Evidence

The report is where the data ends up, and it is not what travels. The moment you hand a result to a CI gate, a dashboard, another team, an auditor or a script somebody wrote two years ago, only a thin middle column moves.

That middle column, he says, is the entire vocabulary the industry has agreed on for what happened when a test ran. The column beside it is everything that vocabulary has no words for.

His worked example is the failing test at two in the morning, flagged as hypothetical. Everything the runner knew existed briefly and was reduced to a status and a stack trace.

He turns that on the trend chart. Twenty builds all green is the line you read in a planning meeting to decide whether quality is improving, and it is a trend of one bit, twenty times over.

Because it carries only that one bit, the chart cannot notice that somebody rewrote the test partway through the sequence. It does not have the vocabulary to notice.

The shape of problem one is a symmetry: the cheapest thing produced, one bit, travels absolutely everywhere, and the expensive part, the one that cost the browser and the compute and the wall clock time, never leaves the room it was born in.

Problem Two: N Times M

Problem two follows from problem one. If the shared format cannot carry the good material, everyone builds their own, which he calls the only rational move. He describes a slide with producers on one side, consumers on the other, and every line between them an integration somebody wrote and somebody maintains forever.

The cost is not the writing, it is the arithmetic. N producers times M consumers. Add one framework and you have added M new integrations; add one consumer and you have added N.

The consequence he calls the actual tragedy of the space. At that price nobody builds the consumer, because it is never worth it, so every framework ships its own dashboard and the dashboards compete with each other.

His sharpest version is that the industry has a great many very smart people competing on reporting when they could be competing on finding bugs.

He names a second bill nobody itemises: what actually makes switching framework expensive. Not the tests, which you can port, and which he prices casually at about a week of annoying work. It is everything wired to one framework’s output, the dashboards, the gates, the analytics.

That is lock-in nobody sold you, accumulated one integration at a time while everyone was busy. He adds that there is not even an official schema for the format everyone standardised on, and that everyone speaks a slightly different dialect of it. He names no framework, standard or vendor anywhere in this section, and offers no evidence for those claims.

Problem Three: Trust Broke

Problem three is the one he says explains why he is giving this talk in 2026 rather than 2016. The trust model broke, and tests became mutable, and both happened fast.

He slows down on the first, calling it the most important sentence in the talk. The old answer to how you know a test is right was that a person wrote it and another person reviewed it, and that review was the artifact. It was never written down anywhere, there was no file, and everyone accepted it because a human being had put their name on it.

Now a model writes it, an agent runs it, and when it breaks overnight an agent repairs it and moves on. His formulation is precise: it is not that people stopped trusting that answer, the thing itself stopped existing.

The second change is mutability. Suites now write themselves and rewrite themselves unattended, which turns whether this is the same test that passed last week from a rhetorical question into one you genuinely cannot answer.

He traces the downstream damage. Flakiness detection works by comparing a test to its own history and trend charts compare build to build, so if the test changed in between you are comparing it to a version of itself that no longer exists anywhere. The comparison still renders, the chart still draws, and it looks valid.

His fix is deliberately unglamorous. A screenshot cannot help, because it tells you what the page looked like rather than what the test was. A hash of the test file fixes it, and he admits it is almost embarrassing how simple that is. He adds that reports designed for human eyes are increasingly read by agents, so verification has to be machine-readable or in practice it will not happen.

Problem Four: Prove It

Problem four is somebody walking in and asking you to prove it. His candidates are an auditor, a compliance review, or just the person who inherits your test suite after you move teams.

What ran, against what build, and with what results are all survivable. He prices those at a decent CI setup and twenty minutes of digging, annoying but fine. This is the point the published description inverts, presenting the easy three as the hard requirement.

Then come the ones that make the room go quiet. The first is not whether you have a dashboard someone can log into. It is whether a person with no access to your system, sitting somewhere else, can reach the same conclusion you did.

The second is to show the evidence for last quarter’s releases, and his answer is that you cannot. The CI logs expired, and the dashboard is mutable, so what it shows today is not necessarily what it said months ago. Screenshots pasted into a wiki page are detached from the run that made them, so they prove nothing.

He is specific about why people misdiagnose this. The data existed, absolutely. Nobody planned to keep it, nobody budgeted the storage, and by the time somebody asked, the retention window had closed.

He pairs that with a one-way ratchet claim, that every year brings more regulation, more complexity and more things you are expected to be able to show. That is his assertion rather than a sourced trend.

Claim Versus Proof

Proof, he argues, is a different category: the material that would let somebody who was not there reach the same conclusion on their own, without trusting you.

He borrows the pattern from three fields that make claims at scale. Finance has ledgers, and a ledger is not a summary. Aviation has flight recorders, and a recorder is not the pilot’s report. Security has audit trails.

In each case there is the claim and, separately, the ground for the claim, and nobody confuses them because in those fields confusing them has serious consequences.

Testing never made that split. The industry built the claim and shipped it everywhere, and nobody built the second thing. His framing of what his team did is that they built the second thing.

He restates it later in machine terms. A report is a rendering of a pack, and the pack is a record. Right now the industry keeps the rendering and throws away the record.

Inside The .evidence Pack

He introduces .evidence as one sealed archive per run: one run, one file. It is versioned, and the moment it is sealed it is marked finalized, which he says is the only authoritative state.

The core layer, L0, is deliberately tiny: a run file, your own test file, and a result file. That alone, he says, is a complete valid pack. In the Q&A he restates the three as a run manifest, your test file, and a structured result carrying the outcome of every step.

The critical property is what the format refuses to do with your test file. It goes into the pack, the format never opens it, it carries it and hashes it, and it does not parse it.

L1 is the captured layer above: logs per step, screenshots, coverage, and a failure record for every failure. He keeps that boundary sharp all the way into the Q&A.

Underneath both is a compatibility rule he says determines whether the format is still useful in five years. A profile only ever adds, it never rewrites the core, and the version number moves only if the meaning of something that already exists changes.

The payoff he claims for that rule is that a reader written this afternoon still works years from now against profiles nobody has thought of yet. That is his design claim, not a demonstrated result.

Three Commands And Refusals

The tool is the reference implementation: it validates a pack against the contract and seals it. Three commands, no more.

Finalize does the derivation, rolls up totals, hashes each definition and seals the archive. Validate is a conformance check whose exit codes do real work: zero if valid, one if not, two if you call it wrong. That means a pipeline can gate on conformance without writing a parser, with no dependencies, in one line of shell.

Merge exists because of how CI actually runs. Shard a run across runners and you get a pack from each, and you want one artifact, so merge combines them under a few declarative rules.

He spends as long on what the tool deliberately does not do. It captures no screenshots and no network data, it does not drive a browser, does not run a test, and has no opinion whatsoever about your framework. The producer or the agent puts the artifacts in, and the tool validates and seals what somebody has already written.

It knows nothing about any framework’s definition format, because it references your test file and hashes it without ever reading it. That refusal, he says, is the entire trick. A validator that understood your tests would need updating for every framework, version, dialect and plug-in forever, permanently out of date and permanently somebody’s problem. One that refuses to look inside never needs updating at all.

His structural argument for that neutrality is that it is not a design preference. A trust artifact, in his words, cannot be owned by a tool that it is judging.

Emitting A Pack Yourself

He describes an emit recipe slide, a few files and commands that produce a sealed pack anyone in the world can validate, and prices the recipe at five lines in his closing ask. It was never run.

Two details, he says, will save you a support ticket. Set the producer name to your tool, because that is how a reader knows what made the pack and how to interpret it.

Second, every step needs an ID, a status, and an ordinal that strictly increases, and the validator rejects duplicates. That ordering is how a reader reconstructs what happened in what order.

The reason the slide is in the deck, though, is the sentence under it: one shape whatever made it. A browser agent, an automated run, an API check, or the internal thing a platform team wrote in Go that nobody outside the company has heard of, all in the same shape.

He frames the ask as an invitation. If you maintain a reporter, this is a rendering target. If you maintain a framework, it is a small number of files. Nobody is being asked to throw anything away, because existing capture formats get reused inside the box and none of them were reinvented.

The detail he says reveals most about the design is that the specification’s own example records an agent as the author rather than a person, and that the repository’s issue templates and contributing guide admit AI agents as filers alongside humans. Tying it back to problem three: if a model wrote the test and an agent repaired it overnight, the record has to be able to say so, because a record that pretends a human did it is worse than no record at all.

The Open-Source Scope

He says the reference implementation is open source, and names Apache 2.0 as part of his closing ask. No release date, version number, package registry or repository URL is stated anywhere in the session, so none is reproduced here.

His pitch is that the code is not the interesting part. The JSON schemas ship in the repository and are the real authority: if the prose and the schema ever disagree, the schema wins.

There is a conformance fixture corpus, which he says is what you would actually need if you are building a producer, because it lets you prove you conform instead of hoping.

There is a decision log of somewhere around 46 numbered records, each carrying the proposition, the options weighed, and what was chosen and why. His example of what it is for is finding out why the verdict field has four values rather than two.

Everything in this section was described rather than opened. Nobody navigated to a repository or displayed a schema, a fixture or a decision record on screen, so this is his account of what the repository contains.

The scope also needs stating precisely, because he draws the line himself and it is easy to blur. The format is open and the validator is open. The viewer is not, and Kane CLI is a commercial product.

The Kane CLI Layer

He is explicit about the relationship. Kane CLI mounts the reference implementation in process and consumes the published package exactly the way anyone else would, with no internal fork, no special build and no private version with extra features. Anything they needed went into the open repository only.

What Kane CLI adds is the captured layer. A screenshot per step, plus a second annotated copy with the element the agent acted on highlighted. He argues this is not cosmetic, claiming that the overwhelming majority of why-did-this-fail questions come down to the agent clicking the wrong thing, which one glance answers. He puts a round percentage on that with no dataset, sample or method behind it, so it is his rhetorical figure rather than a measurement.

Kane packs also carry the full network capture attributed per step rather than one enormous block for the run that you have to line up by timestamp, the browser console per step, a triaged failure record on every failure with root cause and suggested fix, and the agent’s full trajectory.

They also record an environment fingerprint covering the model, the browser, the operating system and the resolution.

Sealing is not opt-in. Every run seals a pack, and he presents it as what a run leaves behind rather than a flag anyone has to remember to set.

His size comparison was stated rather than shown: a pack you build yourself from the recipe is a handful of files and a few kilobytes, while a Kane pack from a comparable run is dozens of entries and several megabytes. Same format, same validator, same seal, with the only difference being how much the producer bothered to capture.

With four minutes left, the host cut in, and the rest was compressed. He described a viewer with four tabs covering the report, test instances, issues and coverage, plus a root cause panel giving plain-English analysis, reproduction steps, console, network and trajectory at the failing step, and a copyable fix you can hand to an agent. He flagged that the coverage tab is requirement coverage rather than code coverage, and that the viewer renders packs produced by Kane CLI.

Q & A Session

The host read out two questions submitted to the Q&A box.

• How do you make agent run results trustworthy to someone who was not there?

  Anubhav Singhmaar: The pack is a sealed file, so it travels, and it validates completely offline: whoever is checking needs no account, no login and no access to anything of yours. The test definition travels inside the pack, so an edit after the fact breaks the hash and the mismatch is visible. And validation exits with a status code, so a machine can check it without a human in the loop. That restates the design guarantee; no external verification, third-party audit or worked example of a tampered pack being caught was offered.

• What counts as evidence in an agentic test run: screenshots, traces, or reasoning logs?

  Anubhav Singhmaar: All three, but let me split that into two layers, because the distinction matters. The format itself requires only a run manifest, your own test file, and a structured result with the outcome of every step. It is kept small on purpose, so a plain API check can produce a pack as easily as a browser agent. Everything the question named lives in the layer above: in a Kane CLI pack, a screenshot per step plus an annotated copy highlighting the element acted on, the network capture attributed to the step that caused it rather than dumped as one block, the console per step, the agent’s full trajectory, and on any failure a triage record with root cause and suggested fix.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.