Batch Runs (testrun)

kane-cli testrun run executes many authored _test.md files as one execution: one summary, one exit code, and one sealed evidence pack for the whole suite.

kane-cli testrun run                                              # every *_test.md under the cwd
kane-cli testrun run tests/checkout_test.md tests/login_test.md   # explicit paths
kane-cli testrun run --tags smoke --parallel 4                    # select by tags, 4 workers

Use testrun when you have a suite of committed tests to run together: nightly regression, pre-merge smoke, release gates. For a single test, kane-cli testmd run is all you need.

Selecting tests

A suite is made of members, one per _test.md file. You either list the files as explicit paths (each must end in _test.md), or give no paths and kane-cli searches the current directory and its subdirectories. Two filters then narrow the list, in this order:

--match <regex>: keeps tests whose path, relative to the project, matches the regex. The path uses your operating system’s separators: tests/app/ on macOS and Linux, tests\app\ on Windows. In cmd.exe, wrap the regex in double quotes, because single quotes are treated as literal characters.
--tags <list>: keeps tests whose tags: frontmatter contains any of the given tags. Matching ignores case. You can repeat the flag or pass a comma-separated list: --tags smoke,checkout and --tags smoke --tags checkout select the same tests.

kane-cli removes duplicates and runs the final list in a stable, repeatable order.

kane-cli testrun run --match 'tests/e2e/.*' --tags smoke

Each path you pass must be a _test.md file. A directory such as tests/app/ is refused with not a *_test.md file. To run every test in a directory, select it with --match '^tests/app/' instead.

Preflight

Before anything runs, kane-cli checks every member. This check is called preflight:

A member does not have to be authored yet. A member with no recording runs as an author member: the agent authors it during the run. Afterwards, the authored and replayed evidence is combined into one published execution. Combining is best-effort: if it cannot complete, the evidence stays split into separate parts rather than being lost.
All members must belong to one org and one project. A testrun is a single execution in Test Manager, so it cannot span projects.

A member can fail preflight for these reasons:

Reason	Meaning	Fix
`org_mismatch`	Belongs to a different organisation than the rest	Check with `kane-cli testmd status <path>`
`project_mismatch`	Belongs to a different project than the rest	Check with `kane-cli testmd status <path>`; run project-by-project

If any member fails preflight, the plan is invalid and nothing runs (exit 2). The failing members are printed to stderr:

error: plan invalid — 2 offending test(s):
  tests/other_org_test.md: org_mismatch
  tests/other_project_test.md: project_mismatch

A member whose authored steps use a {{name}} that has no value still passes preflight. kane-cli prints a warning that lists every such name across the members, with the test files and steps that use it. During the run, the name is typed literally as written, unless an earlier step sets its value first:

warning: 2 variables have no value

  Not in any variables file
    other_key   b_test.md step 1
    shared_url  a_test.md step 1 · b_test.md step 1

    Add them to .testmuai/variables/variables.json

  If {{name}} is literal page text, write \{{name}} to keep it as-is.
  A step that sets a name first binds it; anything else is typed as written. Fill the values to bind them.

testrun run has no --variables flag. To supply values, fill the shared variables file or the member’s own variables: frontmatter. In agent mode (stdin is not a TTY, or you pass --agent), the same information arrives as one warning event with code: "unresolved_variables", right after the testrun_plan event. Each member listed in testrun_plan also carries its own rows. See Unresolved variables for the event shape.

Mobile members

A _test.md with a mobile target: (emulator / simulator) runs as a normal member:

Locally, the suite drives the emulators and simulators on your machine. The machine must be macOS on Apple Silicon, with the mobile setup complete. Choose the device with --device-name / --os-version, using the values exactly as kane-cli devices list --target emulator|simulator prints them, or set device_name: / os_version: in the file.
On the cloud grid (--remote), the suite runs on a virtual device on a HyperExecute macOS host, so you can start it from any machine: Linux, Windows, or a Mac without Xcode or Android Studio. Choose the device from kane-cli devices list --target emulator|simulator --remote. One grid job runs one platform: emulator members on one Android version, or simulator members on one HyperExecute pool. If a member uses a local app build, kane-cli uploads it from your machine before the job starts and passes it to the grid as an APP… id. For everything else, see Remote Runs on the Cloud Grid.

kane-cli testrun run --match '^tests/app/' --device-name "Pixel 7 API 35" --os-version 15            # local emulators
kane-cli testrun run --match '^tests/app/' --remote --device-name "Pixel 7" --os-version 14           # the grid

Running

Flag	Description	Default
`--match <regex>`	Filter candidates by project-relative path regex	none
`--tags <list>`	Keep tests whose frontmatter tags match any listed tag (repeatable or comma-separated)	none
`--parallel <n>`	Worker count	`1`
`--on-failure <mode>`	`continue` \| `fail-fast`	`continue`
`--name <label>`	Run title	derived from the selection
`--dry-run`	Build and validate the plan only; run nothing	off
`--agent`	Print NDJSON on stdout even in a terminal. This is the same stream you get when stdin is not a TTY	off
`--no-adaptive-heal`	Turn off adaptive healing after a replay failure	healing enabled
`--bug-detection <mode>`	`off` \| `stop` \| `continue`. See Configuration	config value
`--headless`	Run Chrome without a visible window	off
`--remote [backend]`	Dispatch the suite to the cloud grid instead of local Chrome or local devices (default backend: `hyper`). Needs `kane-cli plugin install remote-execution`. See Remote Runs on the Cloud Grid	off
`--device-name <name>`	Device for the suite’s mobile members: as `kane-cli devices list` prints it locally, or a grid catalog device with `--remote`	member’s `device_name:`
`--os-version <version>`	OS version for the mobile members (`14`, `17.5`). Alone, it matches any device on that version	member’s `os_version:`
`--username <user>` / `--access-key <key>`	Basic auth instead of OAuth	none

Each desktop worker gets its own isolated Chrome with a fresh temporary profile. Parallel members never share cookies, logins, or tabs, and they never touch your own browser profile. --on-failure controls what a failed member does to the rest of the suite:

continue (default): every member runs, and failures are collected in the summary.
fail-fast: after a failure, no new members start. Members that are already running finish normally.

Ctrl-C stops the run gracefully. No new members start, running members finish, the evidence pack is still sealed, and the run exits 3. Members that never started are reported as skipped. The pack lists every planned member, including skipped and broken ones.

Dry runs

--dry-run prints the exact plan a real run would execute (the selected members, any preflight failures, and the parallelism), then exits without launching anything:

kane-cli testrun run --tags smoke --parallel 4 --dry-run

Exit 0 means the plan is valid. Exit 2 means the plan was refused. A valid plan does not guarantee the run will work: a local dry run stops before authentication and before any browser or device starts. A remote dry run also checks the grid catalog, but it does not upload apps or start a job. A real run can still fail after a valid plan. A dry run also prints the variable warning, so you can fill in missing values before the real run.

Reading results

At the end of a run, you get a suite summary (counts of passed, failed, broken, and skipped members, plus the total duration) and one sealed evidence pack covering every member. The pack is written to .testmuai/evidence/. In a terminal, kane-cli offers to open the pack in the evidence viewer. In CI, it prints the evidence serve hint instead. The pack is also published to your project’s execution history in Test Manager. --name sets the run’s title, which helps you tell nightly runs apart in the dashboard.

Exit codes

Code	Meaning
`0`	All members passed.
`1`	At least one member failed or broke.
`2`	Usage error, invalid plan (preflight failures), or auth error. Nothing ran.
`3`	Cancelled (Ctrl-C).

Remote runs

With --remote, the same selection runs as one HyperExecute job. Web suites run in Chrome on a grid macOS runner, and mobile suites run on a grid emulator or simulator. Each member runs as its own headless testmd run on the grid, and --parallel N spreads the members across N grid runners. The job link, the recordings, and the evidence pack come back to your project. You need a plan with HyperExecute (macOS runners) and the remote-execution plugin. Nothing else is needed on your machine, not even Chrome.

kane-cli plugin install remote-execution
kane-cli testrun run --tags smoke --remote --dry-run          # resolve everything, dispatch nothing
kane-cli testrun run --tags smoke --remote --parallel 4       # a web suite on 4 grid runners
kane-cli testrun run --match '^tests/ios/' --remote --device-name "iPhone 15" --os-version 17.5

--dry-run also runs the remote preflight, so the selection is checked against the grid before any job is created. Web tests and device tests cannot share one job, so run them as two separate suites. --remote is different from --ws-endpoint, which connects a remote browser to a run that still executes on your machine. For the full guide (prerequisites, how a job runs, web suites, the device catalog, app rules, what one job can hold, and the events), see Remote Runs on the Cloud Grid.

Using testrun in CI

kane-cli testrun run --tags smoke --parallel 4 --headless --on-failure fail-fast

The exit code passes or fails the pipeline. Save .testmuai/evidence/*.evidence as a CI artifact: it is a single file per suite run that anyone can open in the viewer. With --remote, the CI runner needs no Chrome or mobile tooling at all. For complete recipes, see CI/CD.

For agents: NDJSON events

In agent mode (or whenever stdin is not a TTY), testrun run writes its own typed NDJSON events to stdout: testrun_plan, testrun_start, testrun_member_start, testrun_member_end, testrun_investigations_wait, testrun_evidence_ingest, testrun_summary, and finally testrun_done, which ends the stream. Stop parsing at testrun_done. A --remote run wraps that stream in remote_* events (remote_start, remote_device, remote_dispatched, remote_error, remote_done). See Remote Runs. The full event schema ships with the kane-cli agent skill.

Execution constraints

A local suite that contains any mobile member must use --parallel 1; larger values are refused. Isolated Chrome workers apply to desktop members only. Remote suites can run members in parallel on the grid with --parallel N. Adaptive healing is on by default. When a replay fails, the runner retries smaller and smaller replay windows (up to three), then re-authors the authorable steps. --no-adaptive-heal turns this off. The --retry and --retry-count flags are accepted but have no effect: they only print a notice. Adaptive healing handles failed replays. Steps marked replay-only keep their recordings even during healing. kane-cli decides whether to print NDJSON by looking at stdin, not stdout. For automation started from a terminal, run kane-cli testrun run <paths> < /dev/null. A dry run validates the plan only, not authentication or browser and device readiness. Always check the process exit code, because some failures end without a normal completion event.

Next steps

Remote Runs on the Cloud Grid: --remote, mobile suites from any machine.
Evidence packs: what’s in the pack and how to view it.
Writing test.md files: the file format, including tags:.
Running test.md files: single-test runs, replay, and flags.
CI/CD recipes: pipeline patterns.

​Selecting tests

​Preflight

​Mobile members

​Running

​Dry runs

​Reading results

​Exit codes

​Remote runs

​Using testrun in CI

​For agents: NDJSON events

​Execution constraints

​Next steps

Selecting tests

Preflight

Mobile members

Running

Dry runs

Reading results

Exit codes

Remote runs

Using testrun in CI

For agents: NDJSON events

Execution constraints

Next steps