Verification Bench, a benchmark for AI testing agents
Every night, Kane CLI and two coding agents driving Playwright MCP get the same 25 workflows on four live TestMu AI apps. We count the tests each one authors and how often it reports a pass the page contradicts.
Authoring and false passes: Kane CLI, Claude Code, Codex
Averages over the nights run in October 2026. Rows marked worked out come from the rows above them, using the unrounded averages (75.2% for Kane CLI, 65.6% for Claude Code).
| Capability | Kane CLI | Claude Code and Codex, with Playwright MCP |
|---|---|---|
| Workflows authored, all 25 | 75% | Claude Code 66%, Codex 50% |
| Workflows authored, cloud execution console (5) | 64% | 0% for both |
| Workflows authored, other 20 (worked out) | About 78% | Claude Code about 82%, Codex about 63% |
| False pass rate, all runs | 0.8% | Claude Code 16.8%, Codex 13% |
| False pass rate, other 20 workflows (worked out) | 1% or less | Claude Code about 7%, Codex about 7.5% |
| Reported passes that were wrong | 1% | 20% for both |
Kane CLI authored 75% of 25 workflows a night, on average
Averaged over the October nightly runs, Kane CLI authored 75% of the 25 workflows, about 18.8 a night. Claude Code with Playwright MCP authored 66% and Codex with Playwright MCP 50%. On its best night Kane CLI authored 84%; Claude Code's best night was 68%.
All of Kane CLI's lead over Claude Code, and about half its lead over Codex, comes from the cloud execution console, where Kane CLI authored 64% of the five workflows and neither coding agent authored any.
On the other 20 workflows, Claude Code authored about 82% and Kane CLI about 78%. Claude Code never authored eight of the 25 workflows on any night; Kane CLI authored each of them at least once.
0.8% of Kane CLI runs were false passes a grader caught
Kane CLI reported a false pass, every step green while the page showed otherwise, on 0.8% of runs. Claude Code with Playwright MCP did on 16.8% and Codex with Playwright MCP on 13%.
Two thirds of Claude Code's false passes came from the cloud console, where it reported one on 56% of runs and Codex on 35%. On the other 20 workflows the rates were about 7% for Claude Code, 7.5% for Codex and 1% or less for Kane CLI.
These rates count only the contradictions the grader found, so the true rate may be higher for every setup. The grader confirmed 46% of Kane CLI's checks and found 0.9% contradicted; it could neither confirm nor contradict the rest.
Checks confirmed and contradicted: Kane CLI vs Claude Code
This page shows check-level figures for Kane CLI and Claude Code only; the Codex figures were not in the data we published from.
| Capability | Kane CLI | Claude Code + Playwright MCP |
|---|---|---|
| Checks the grader confirmed on the page | 46% | 31% |
| Checks the page contradicted | 0.9% | 5.7% |
| Checks it could neither confirm nor contradict (worked out) | 53.1% | 63.3% |
Kane CLI used about 29% less time a test than Claude Code
On tests that both Kane CLI and Claude Code with Playwright MCP authored on the same night, Kane CLI took 71.5% of Claude Code's time, about 1.4 times faster. It finished first in 61% of those head-to-head tests.
Codex with Playwright MCP has no authoring-time figure in this edition.
Per-workflow time savings for Kane CLI over Claude Code
Five workflows from the head-to-head comparison against Claude Code with Playwright MCP, measured only on nights when both setups authored the test.
Manual test cases land in the right folders
67% less time, 3.0 times faster.
If-Else and While Loop control flow
67% less time, 3.0 times faster.
New authoring experience, two objectives
63% less time, 2.7 times faster.
Edit a test case, reload, mint a new version
45% less time, 1.8 times faster.
Rename and delete folders
42% less time, 1.7 times faster.
61% of head-to-head tests
Kane CLI finished first in 61% of the tests both setups authored on the same night.
25 workflows on four live TestMu AI apps
Each workflow is a multi-stage objective in natural language with checks the agent must prove on the page: 188 stages and about 270 checks a night. The apps are TestMu AI's own products.
Cloud execution console
Jobs, projects, secrets and a generated run config: 5 workflows, 38 stages and about 100 checks a night.
Test management app
Folder trees nested ten levels deep, test cases, versions, and If-Else and loop control flow.
AI agent workspace
An agent driving another agent: instructions typed in, then steps moving from queued to running to done.
Real Device Cloud web console
Live device previews and device boots in the Real Device Cloud web console.
Share of cloud console app workflows each agent authored
Authored share for four of the five cloud execution console workflows. An example check: the generated config carries the exact concurrency, retry and command values entered.
| Capability | Kane CLI | Claude Code and Codex, with Playwright MCP |
|---|---|---|
| Generate a run config from a form | 100% | 0% for both, with a false pass on every run |
| Projects: search, scoped jobs, new project validation | 100% | 0% for both |
| Jobs list: filters, clear all, search | 60% | 0% for both |
| Project secrets: add, edit, delete | 40% | 0% for both; Claude Code reported a false pass on every run |
PRISM 2026-Q4 scores for Kane CLI v0.8.17
PRISM scores frontend testing tools on 50 scenarios in 10 categories. Kane CLI v0.8.17 placed first of seven entries as a vendor submission: TestMu AI wrote its tests and PRISM ran them on held-out seeds.
76.7 overall (RPS-Index 0.7674)
The average across 50 scenarios of each scenario's reliability times perception times intent, over six seeds.
88.7 reliability
Mean reliability (R): each click before the page hydrates, on a covered target or during a layout shift costs points.
86.7 perception
Mean perception gate (S): whether the tool used the right control while a person could see it, even after its label or form changed. The highest of the seven entries.
77.9 intent
Mean intent alignment (I), from 0 to 1. A false heal or an action the test never asked for sets a scenario's I to zero.
9 of 50 false heals
Green results over broken flows. Four of the other six entries recorded fewer.
3 intent and 0 integrity violations
In three scenarios Kane CLI acted on a control the test never asked for. PRISM recorded no integrity violations.
PRISM 2026-Q4: Kane CLI vs scripted Playwright tests
Playwright v1.60.0 is PRISM's scripted baseline: plain Playwright tests that PRISM ran on the same held-out 2026-Q4 seeds, with no agent in the loop. PRISM writes ranked tests from the published specs without seeing the pages, and notes that this puts a scripted suite at a disadvantage. Kane CLI's index is 1.67 times Playwright's, and Playwright recorded one fewer false heal and one fewer intent violation. Its scores are on Playwright's PRISM page, and we compare the two as tools on Kane CLI vs Playwright.
| Capability | Kane CLI v0.8.17 | Playwright v1.60.0, scripted |
|---|---|---|
| RPS-Index, overall | 0.7674 | 0.4599 |
| Mean reliability (R) | 0.8871 | 0.6409 |
| Mean perception gate (S) | 0.8667 | 0.6900 |
| Mean intent alignment (I) | 0.7787 | 0.5200 |
| Categories led, of 10 | 9 | 1, Lying DOM |
| False heals, of 50 scenarios (lower is better) | 9 | 8 |
| Intent violations (lower is better) | 3 | 2 |
| Integrity violations | 0 | 0 |
Scores in 10 PRISM categories: Kane CLI vs Playwright
Category index from 0 to 1, five scenarios per category. Kane CLI led in nine of ten categories, by 0.40 or more in six, and its 0.4000 on multi-step logic was the highest any entry scored there. Every entry is on the PRISM 2026-Q4 board.
| Capability | Kane CLI v0.8.17 | Playwright v1.60.0, scripted |
|---|---|---|
| Hydration and timing gap | 1.0000, 0 false heals | 0.4395, 1 false heal |
| Canvas and WebGL blind spot | 0.7000, 0 false heals | 0.2000, 0 false heals |
| Accessibility and shadow DOM obstructions | 0.6667, 2 false heals | 0.2000, 1 false heal |
| Non-deterministic and generative UI | 0.6200, 1 false heal | 0.2000, 1 false heal |
| Affective and cognitive load | 1.0000, 0 false heals | 0.6000, 0 false heals |
| Peak complexity and multi-step logic | 0.4000, 1 false heal | 0.0000, 2 false heals |
| Dynamic data and network artifacts | 0.9000, 0 false heals | 0.7000, 1 false heal |
| React-specific state and race conditions | 0.8333, 2 false heals | 0.6600, 1 false heal |
| Visual occlusion and layout shifts | 0.8874, 1 false heal | 0.8000, 0 false heals |
| Lying DOM | 0.6667, 2 false heals | 0.8000, 1 false heal |
6 rules every Verification Bench run follows
Same seeded account
Each setup signs in with the same seeded account in the same live app, so every run starts from the same state.
Same browser limits
Kane CLI drives real Chrome over the Chrome DevTools Protocol. Claude Code and Codex drive the same class of browser through Playwright MCP, with the same session limits.
Same brief
Every setup gets the same natural-language objective, its stages and the checks it must prove on the page.
Same time limit
Each workflow has the same time limit, and no run is retried.
A separate grader
A grader built by TestMu AI checks each claimed result against the page. A run whose reported pass the page contradicts counts as a false pass, whichever setup made it.
Nightly runs
The suite runs nightly. Headline figures are averages across the nights run in October 2026, and best-night figures are labeled.
Limits of the Verification Bench results
TestMu AI builds Kane CLI and designed, ran and graded this benchmark on its own live products. Kane CLI shares its engine with KaneAI and syncs with Test Manager, and the brief format (objective, stages, checks) is the input Kane CLI takes natively.
The 25 workflows are not public, so a third party cannot rerun them. This edition breaks down the cloud console but not the other three apps.
The bench measures test authoring and whether reported passes hold up, not exploratory bug-finding. It records no spend per run, so time is its only efficiency figure.
It leaves out the grader's model and prompt, the time limit, who wrote the briefs, the Claude Code and Codex model settings, the Kane CLI version each night, the night count, raw counts and spread, and true and false fail rates.
PRISM's 2026-Q4 round was its first ranked round, and Kane CLI led the second entry by 0.0027.
Questions about the Verification Bench method
It measures how many of 25 multi-stage workflows each setup turns into a runnable test within a fixed time without the grader contradicting it, and what share of its runs are false passes.
A workflow counts as authored only when every stage passed, the grader found nothing on the page that contradicts it, and a runnable test came out the other end.
A run the agent reports as passed while the grader finds the page contradicts it, for example when the text it claims to have checked never appeared. The rate is a share of all runs, so lower is better.
Authored share is how much of a test backlog an agent turns into runnable tests, the false pass rate is how many of those tests you can trust, and time is what they cost in wall-clock hours. An agent that authors many tests but reports false passes adds tests that hide bugs.
As wall-clock time to an authored test, and only on tests that both Kane CLI and Claude Code with Playwright MCP authored on the same night.
Claude Code and Codex can check the code they write by driving Playwright MCP themselves. Kane CLI is the verification step those agents can call instead: the agent writes the code, and Kane CLI runs the flow in a real browser and returns an evidence-backed pass or fail. In this benchmark Kane CLI ran each brief directly. See how that works for Claude Code app testing and Codex app testing.
One person maintains PRISM, a public benchmark whose scoring code is open and whose sandbox is private, so a third party cannot reproduce its runs. In 2026-Q4, five of the seven entries were baselines PRISM wrote and ran itself. Kane CLI was a vendor submission: TestMu AI wrote its tests and PRISM ran them on its own held-out seeds. PRISM lists the SHA-256 de2893dc61be73052922844151750597663026dc13a992d8740480f8616379b2 for that run's archive, generated on 26 September 2026. Our Kane CLI PRISM benchmark write-up has the details.
Run a representative set of flows several times during a pilot, compare the results against a human or existing-automation baseline, and report four numbers separately: true pass, true fail, false pass and false fail. That gives a reliability figure for your own app rather than a vendor benchmark. Start from the Kane CLI install.
Hand browser checks from your coding agent to Kane CLI
Give Claude Code, Codex or Cursor the Kane CLI guide and it installs Kane CLI, opens the sign-in for you and runs browser checks on its own.
Point your agent to: testmuai.com/kane-cli-agent/AGENTS.md