Running test.md Files

This page covers the kane-cli testmd command family: how to run a test, what the flags do, how replay works, what shows up in the output directory, and what the exit codes mean. To learn the _test.md file format, see Writing test.md files. To reuse flows across tests, see Composing tests with @import.

Commands

kane-cli testmd run <path>          # run a test
kane-cli testmd list                 # list *_test.md files in the current directory
kane-cli testmd status <path>        # show recorded status for a test
kane-cli testmd delete <path>        # delete a test and its output directory
kane-cli testmd export <path>        # regenerate code export from existing recordings
kane-cli testmd sync <path>          # push the test bundle (test + imports + outputs) to the cloud

To run many tests as one execution, selected by path or tags:, use kane-cli testrun run. In a TTY, kane-cli testmd with no subcommand opens an interactive picker that walks the current directory and lets you pick a _test.md to run.

`kane-cli testmd run`

The main command. Point it at a _test.md file and it runs:

kane-cli testmd run amazon_test.md

You can run files anywhere on disk by giving an absolute path or a path relative to your shell’s working directory:

kane-cli testmd run ./tests/e2e/checkout_test.md
kane-cli testmd run /Users/me/project/tests/checkout_test.md

Flag reference

Every flag accepted by kane-cli testmd run:

Flag	Type	Default	Description
`--headless`	flag	off	Launch Chrome without a visible window. Applies to this run only; there is no persistent setting. Same as `headless: true` in frontmatter.
`--max-steps <n>`	integer	`50`	Maximum agent reasoning steps per test step.
`--timeout <s>`	integer	none	Time limit for each step, in seconds; the step is stopped when it runs out.
`--url <url>`	string	frontmatter `url:` / config `default_url`	Start URL for the first step. Overrides the frontmatter `url:` key and the configured `default_url`; bare domains are normalized to `https://`.
`--allow-missing-url`	flag	off	Non-TTY only. If the first step has no start URL, start from the browser’s current page instead of failing. `url:` / `--url` are still used when present.
`--cdp-endpoint <url>`	string	none	Reuse an external Chrome over CDP.
`--ws-endpoint <url>`	string	none	LambdaTest / Playwright WebSocket endpoint.
`--global-context <file>`	path	`~/.testmuai/kaneai/global-memory.md`	Override the global context file.
`--local-context <file>`	path	`<cwd>/.testmuai/context.md`	Override the local context file.
`--variables <json>`	JSON string	none	Inline variables JSON for this run.
`--variables-file <path>`	path	none	Load variables from a JSON file.
`--session-context <json>`	JSON string	none	Context from earlier runs, loaded for the agent before the run starts.
`--username <user>`	string	profile	Basic-auth username. Uses basic auth instead of OAuth for this run.
`--access-key <key>`	string	profile	Basic-auth access key. Uses basic auth instead of OAuth for this run.
`--mode <name>`	`action` \| `testing`	profile or `testing`	Agent behaviour on auth walls and error pages. See Run mode.
`--agent`	flag	on automatically when stdin is not a TTY	Plain NDJSON output, with no interactive terminal UI. Use this in CI.
`--code-export`	flag	config (`true` by default)	Generate Playwright code after the run. See Code export.
`--code-language <lang>`	`python` \| `javascript`	`python`	Code-export target language.
`--skip-code-validation`	flag	on	Skip the validation pass that runs after code is generated.
`--no-skip-code-validation`	flag	off	Run the validation pass after code is generated.
`--name <name>`	string	none	Persist a session name. Must match `[a-zA-Z0-9_-]+`.
`--on-lock-conflict <mode>`	`readonly` \| `fail` \| `wait`	none	Policy when another user holds the lock. See Lock conflicts.
`--push`	flag	off	Always commit at the end of the run.
`--no-adaptive-heal`	flag	healing enabled	Turn off adaptive healing after a replay failure. See Replay policy and completion.
`--author`	flag	off	Re-author all authorable steps; steps marked replay-only still replay.
`--bug-detection <mode>`	`off` \| `stop` \| `continue`	config value (`off`)	Detect product bugs while authoring: `stop` halts the run on a confirmed bug; `continue` records it and keeps going. Overrides `config set-bug-detection`. Applies to authoring steps only; replay failures are always investigated. See Configuration.

Most flags have a frontmatter key with the same name, written with underscores. When both are set, the CLI flag wins. The exception is variables: values in the file win. See Writing test.md files. Before any step runs, kane-cli checks every {{name}} used by the authored steps against the file’s own variables: frontmatter, the variable files, --variables-file, and --variables. A name with no value prints a warning that names the variable, the file where its value belongs, and the step. The run then continues, and the name is typed literally as written unless an earlier step sets its value. Replayed steps are not checked. See Before a run.

How a run works

A run proceeds in four phases:

Parse: kane-cli reads the _test.md file (and any helper it imports) and validates frontmatter, step bodies, and the @import graph. Parse errors abort here with exit 2; no browser is launched.
Decide replay vs author per step: for each step, kane-cli decides whether to replay a cached recording or author a fresh one. See Replay vs author.
Execute: kane-cli launches Chrome and works through every step in order. Each step either replays its cached recording or asks the agent to figure it out.
Finalize: kane-cli writes Result.md, saves recordings, uploads the run to Test Manager, and (if enabled) generates code export.

Each step runs in its own runner process, but all steps share one browser, so cookies, login state, and tabs carry over from step to step.

Replay vs author

The first time you run a _test.md, every step is authored: the agent works out how to satisfy the objective and a recording is saved next to the test. On every later run, kane-cli tries to replay the saved recording for each step, with no agent involved and no LLM cost, so the run is much faster. An authorable step replays only when all of the following are true. (Steps marked replay-only always replay, so these conditions do not apply to them.)

A recording exists for it on disk.
The step’s objective text has not changed since the recording.
The step’s yaml block has not changed since the recording.
No earlier step in the same file invalidated it (see below).

If any of those fail, the step authors again.

Edits cascade to the rest of the file

Editing one step re-records that step and every authorable step after it in the same file. Steps marked replay-only still replay. This is because each step starts where the previous step left off (same browser, same URL, same logged-in state). If you change step 3, the page state that step 4 expects may be different, so replaying step 4 from cache is not safe. As a result, a small text fix at the top of a test can re-author every authorable step in the file. If you are iterating on a test and want only the last step to re-record, edit that last step.

Forcing a fresh recording

To author everything again:

--author: skips the replay decision for one run. Every authorable step is authored again; steps marked replay-only still replay.
Keep the output recordings for steps marked replay-only. Deleting them does not make those steps author again.

Handling replay failures

A replay can fail if the site has changed: a button moved, the page is slower than expected, or an extra modal appears. The healing policy controls recovery:

Adaptive healing is on by default. The runner retries smaller and smaller replay windows (up to three), then re-authors the authorable steps. Pass --no-adaptive-heal to turn it off. See Replay policy and completion.

Every failed replay is also investigated automatically. A failure record (the error, the page state at the moment of failure, and links into the step’s console and network logs) is written into the run’s evidence pack, so you can see why the replay broke, not just that it did.

Replays don’t need a project or folder

When every step of an already-authored test replays from cache, you do not need a Test Manager project or folder configured. kane-cli shows no interactive picker and creates no project: the test already records where it belongs, and the replay runs with that identity. Authoring runs (first runs, edited steps, --author) still resolve a project and folder.

Run mode

The --mode flag controls how the agent handles authentication walls, blocked pages, and error pages. The same setting can be put in frontmatter as mode: (root-only).

testing (default): the agent treats those pages as part of the run and continues. This lets negative tests (for example, “verify the error message is shown”) work, because the run does not stop at the first error page.
action: the agent stops on auth, blocked, and error pages so a person can step in before the run continues.

testing is the right default for automated suites; action is for one-off authoring sessions where you want manual control.

The output directory

A successful run writes everything it needs to replay next time into output-<stem>/ next to the test file, where <stem> is the filename without _test.md:

amazon_test.md
output-amazon/
  Result.md
  .internal/                     # cached recordings, do not edit by hand
  playwright-python-code/        # only if code export is enabled

Result.md is the human-readable run report. .internal/ holds the recordings kane-cli replays on the next run. The whole output-<stem>/ directory is safe (and recommended) to commit to git: it makes your tests reproducibly replayable on any teammate’s machine and on CI. For tests that @import helpers, kane-cli also writes one helper-output-... directory per call site next to the helper file. See Composing tests with @import.

The evidence pack

Besides the output directory, every testmd run seals an evidence pack (screenshots, per-step console and network logs, and failure records for the whole run) and copies it into <cwd>/.testmuai/evidence/. In a terminal, kane-cli offers to open it in the browser viewer at the end of the run. Replayed runs also publish their pack to your project’s execution history in Test Manager. Do not commit .testmuai/evidence/ to git. Packs are run artifacts, not test sources.

`Result.md`

Result.md is generated on every run. It is intended to be read in a browser or editor. Drop it into any Markdown viewer. The file begins with frontmatter:

---
test: ../amazon_test.md
status: passed
started: 2026-05-13T06:22:43.641Z
duration_s: 78
session_id: 1de66066-fc38-4ed4-9427-f28b2e081171
---

Followed by one entry per root-level step:

## Open Amazon ✓ passed (3s)
Open https://www.amazon.com.

## Search ✓ passed (15s)
Type "wireless headphones" into the search box and submit.

## Add to cart ✗ failed (12s)
Click "Add to Cart" on the product page.

Step status icons:

Icon	Meaning
`✓ passed`	The step completed successfully.
`✗ failed`	The step failed and the run did not continue.
`⏭ skipped`	The step did not run because an earlier non-optional step failed.
`(optional)`	Suffix added when a step marked `optional: true` failed but the run continued.

When an @import step fails, the badge points at the failing leaf inside the helper, e.g. ✗ failed (via @import ./helpers/login.md — at sub-step 2 → 3, 8s → ./helpers/helper-output-login-checkout-3). The sub-step chain walks from the import down to the actual failing step.

Exit codes

Code	Meaning
`0`	Run passed (or the file had no steps).
`1`	At least one non-optional step failed.
`2`	Error before or during the run: parse error, auth error, organisation mismatch, Chrome launch failure, invalid flag, lock conflict with `--on-lock-conflict fail`.
`3`	Cancelled (Ctrl+C confirmed) or timed out (including `--on-lock-conflict wait` timing out).

Use these to gate downstream CI steps.

Lock conflicts

Each test in Test Manager has a single-writer lock so two engineers cannot author the same test at the same time. When you run a test and someone else already holds the lock, --on-lock-conflict decides what to do:

Policy	Behaviour	Exit
`readonly`	Proceed in replay-only mode. No commit, no upload. Useful when you just want to verify a test still passes.	`0` on success
`fail`	Abort immediately. Use this in CI to flag contention rather than waiting.	`2`
`wait`	Block until the lock releases or times out. Use this for serialised CI lanes.	`0` on success, `3` on timeout

The lock is acquired before Chrome launches, so a fail policy aborts with zero side effects.

Code export

kane-cli testmd run can generate runnable Playwright code from a successful run. Enable it via frontmatter:

code_export: true
code_language: "python"   # or "javascript"

…or for a single run, with flags:

kane-cli testmd run amazon_test.md --code-export --code-language python

The generated project is written to output-<stem>/playwright-<lang>-code/. Code export runs only after a successful Test Manager upload, so it is only useful when a project and folder are configured. For the configurable options (default language, validation toggle, persistent enable from the TUI), see Configuration.

Running in CI

A CI-friendly invocation:

kane-cli testmd run ./tests/checkout_test.md \
  --agent \
  --headless \
  --on-lock-conflict wait

--agent: plain NDJSON to stdout, with no interactive terminal UI. It turns on automatically when stdin is not a TTY, but pass it explicitly for clarity.
--headless: Chrome runs without a window.
--on-lock-conflict wait: block instead of failing if a teammate is editing the same test.
Adaptive healing recovers from replay failures by default; pass --no-adaptive-heal to turn it off.

In a non-interactive run (stdin is not a TTY), there is no one to answer an interactive ask_user prompt, so kane-cli disables it: a step that would otherwise wait for input fails cleanly instead of blocking forever. Write test steps that do not depend on mid-run prompts when running in CI. If the runner cannot have Chrome at all, run the tests as a suite on the cloud grid instead. kane-cli testrun run --match '^tests/' --remote executes every testmd run on a HyperExecute macOS runner and returns the recordings and the evidence pack. See Remote Runs. Capture exit code in a shell script:

kane-cli testmd run ./tests/checkout_test.md --agent --headless
status=$?
if [ "$status" -eq 0 ]; then
  echo "passed"
elif [ "$status" -eq 1 ]; then
  echo "failed; check Result.md"
  exit 1
else
  echo "error or cancellation (exit=$status)"
  exit 1
fi

The runner streams NDJSON events to stdout in --agent mode. Each line is a JSON object you can parse in a downstream step.

`kane-cli testmd list`

Walks the current directory and prints every *_test.md file it finds:

kane-cli testmd list

Useful for sanity-checking that your tests live where you think they do, and for piping into other commands.

`kane-cli testmd status <path>`

Shows the Test Manager identity of a recorded test: the project, the folder, the testcase ID, and whether the local recordings are in sync with the last upload.

kane-cli testmd status amazon_test.md

If you have never run the test, status reports it as not yet recorded.

`kane-cli testmd delete <path>`

Removes the test source and its output-<stem>/ directory:

kane-cli testmd delete amazon_test.md

This is a local-only delete. It does not remove the test from Test Manager. Use it when you have abandoned a test or want to start over from scratch.

`kane-cli testmd export <path>`

Regenerates the code export from existing recordings without re-running the test:

kane-cli testmd export amazon_test.md --language python

This is faster than a full run because no browser is launched; it reuses the last recordings.

`kane-cli testmd sync <path>`

Pushes a test’s replay bundle to the cloud: the _test.md itself, every helper it @imports, and the replay-required outputs (Result.md, recordings). The bundle is tied to the test’s last commit.

kane-cli testmd sync ./tests/checkout_test.md

Flag	Description
`--username <user>`	Basic-auth username (skips OAuth)
`--access-key <key>`	Basic-auth access key

You rarely need to run this yourself: after every successful authored commit, kane-cli pushes the same bundle automatically. Use the manual command to re-sync a test when the automatic push failed (for example, because the network dropped at the end of a run).

Replay policy and completion

Adaptive healing is on by default. After a replay failure, the runner tries up to three smaller and smaller replay windows, then fully re-authors the authorable steps. --no-adaptive-heal turns this recovery off. The --retry and --retry-count flags are accepted but have no effect: they only print a notice, and they do not turn on healing or change its fixed number of attempts. Steps whose heading is marked @db, @api, @js, @smartui, @network_query, or @network_assertion are replay-only. They still replay under --author, when an earlier step changes, and during adaptive healing. If a replay-only step’s recording is missing, kane-cli cannot re-author it from the marker: restore the recording, or recreate it with the workflow that originally produced it. Do not delete these recordings to force authoring. Adding a marker to a heading does not create the operation. Structured control flow uses heading markers that must be balanced: @if, @elif, @else, @end-if, @while, @end-while. An @else must be the last branch in its conditional, and each end marker must match the type of block it closes. These markers are separate from conditionals written in natural language. The markers themselves are not included in the step body hash. A step can have only one replay-only marker, and an @import step cannot also be marked replay-only. Under --agent, wait for both the test_md_done event (which carries the file-level overall_status, duration_s, session_id, and an optional share_url) and the process exit. Individual run_end events do not mean the file is complete.

Next steps

Writing test.md files: file format, frontmatter, steps, variables.
Batch runs with testrun: run many tests as one execution with one evidence pack.
Evidence packs: what every run captures and how to view it.
Composition with @import: reusable helpers across tests.
Configuration: persistent settings: Chrome profile, window size, code export defaults.
Test Manager integration: what happens during the upload.

​Commands

​kane-cli testmd run

​Flag reference

​How a run works

​Replay vs author

​Edits cascade to the rest of the file

​Forcing a fresh recording

​Handling replay failures

​Replays don’t need a project or folder

​Run mode

​The output directory

​The evidence pack

​Result.md

​Exit codes

​Lock conflicts

​Code export

​Running in CI

​kane-cli testmd list

​kane-cli testmd status <path>

​kane-cli testmd delete <path>

​kane-cli testmd export <path>

​kane-cli testmd sync <path>

​Replay policy and completion

​Next steps