How to Write Kane CLI Tests in Markdown

testmd lets you write browser tests as Markdown files (_test.md) and commit them to your repo. On the first run, the AI agent authors each step and saves a recording. On every subsequent run, each step replays from cache with no LLM cost and much faster execution. Commit the test file and its recordings to git so teammates and CI can re-run the same tests without re-authoring.

Quick Start

Step 1: Create a test file. The filename must end in _test.md:

---
mode: testing
---

# Amazon search

## Open Amazon
Open https://www.amazon.com.

## Search for headphones
Type "wireless headphones" into the search box and submit.
Verify at least one product result is visible.

Step 2: Run the test:

kane-cli testmd run amazon_test.md --agent

On the first run, the agent authors each step and caches the recording. On every later run, the steps replay from cache instantly.

Always combine --agent with --headless in CI/CD environments to avoid display server errors.

When to Use testmd vs run

kane-cli run is one-shot. It runs an objective, uploads results, and exits. It is ideal for quick, one-off verifications like checking if a page loads correctly or extracting a value from a live site. kane-cli testmd run is for tests you want to persist. Use it when you are building a login flow smoke test, a regression suite, or any test you plan to re-run across builds. The test file lives in your repo, recordings are cached and committed alongside it, and every subsequent run replays from cache without consuming LLM credits. Teammates and CI pick up the same recordings and replay them identically. If you run an ad-hoc objective with kane-cli run and later decide you want to keep it, use the --name flag to save it as a _test.md file (see Recording a Test from a Live Session below).

File Format

A _test.md file has four parts in order:

YAML Frontmatter

Configuration between --- markers at the top of the file:

---
mode: testing
max_steps: 30
headless: true
---

Supported keys:

Key	Scope	Description
`mode`	root	`testing` (default) or `action`. Testing mode pushes through auth walls for negative-test assertions. Action mode halts on auth walls.
`url`	root only	Start URL for the test’s first step. Bare domains are normalized to `https://`. Overridden by the `--url` flag; falls back to the configured `default_url`. See Default start URL.
`tags`	root only	Labels for selecting tests in batch runs. Accepts a YAML list, a bracketed inline list (`tags: [smoke, checkout]`), or a bare comma string (`tags: smoke, checkout`). Tags are trimmed, lowercased, and de-duplicated. Shown by `kane-cli testmd list` and the interactive picker; selected with `testrun --tags`; recorded in the run’s evidence pack.
`max_steps`	root + step	Max agent reasoning steps per step. Default: `30`
`timeout`	root + step	Hard kill per step, in seconds
`headless`	root	Run Chrome with no visible window
`variables`	root + step	Variables with `{{name}}` syntax, same format as Variables & Context
`session_context`	root + per-step	Pre-loaded prior-run context for the agent.
`code_export`	root + step	Generate Playwright code after the run
`code_language`	root + step	`python` or `javascript` for code export
`global_context` / `local_context`	root + step	Inline Markdown or file path for agent context
`target`	root	Where the test runs: a browser transport (`chrome`, the default, `cdp`, or `ws`) or a mobile target (`emulator` or `simulator`, a virtual Android or iOS device, locally on macOS Apple Silicon or on the cloud grid with `testrun run --remote`). See Mobile Target.
`app`	root	Mobile only. The app under test: a build path (emulator `.apk`, simulator `.zip`) or an uploaded `APP…` id. Required with a mobile target, rejected with a browser target.
`no_reset`	root	Mobile only. Keep the app’s existing state between runs instead of resetting it.
`device_name`	root	Mobile only. The default device for this test, as `kane-cli devices list --target <kind>` prints it (add `--remote` for the grid catalog). Overridden by `--device-name`.
`os_version`	root	Mobile only. The device’s OS version (`14`, `17.5`). Overridden by `--os-version`. Required alongside `device_name`.
`chrome_profile`	root only	Named Chrome profile under `~/.testmuai/kaneai/chrome-profiles/`.
`cdp_endpoint`	root only	Reuse an external Chrome over CDP.
`ws_endpoint`	root only	LambdaTest / Playwright WebSocket endpoint.
`on_lock_conflict`	root only	Policy when another user holds the lock on this test in Test Manager. See Lock conflicts.

Mobile Target

target: also accepts the two mobile values, emulator for a virtual Android device and simulator for a virtual iOS device, with the app under test as its own root key:

---
target: emulator             # emulator (Android) | simulator (iOS)
app: ./builds/app-debug.apk  # or an APP… id from `kane-cli apps list --target emulator`
device_name: Pixel 7         # optional default device (with os_version)
os_version: "14"             # optional; `--device-name` / `--os-version` override both
no_reset: false              # optional
---

target: emulator runs on an Android emulator, simulator on an iOS simulator. The platform never appears separately, the target implies it.
app: the app under test, required with a mobile target and rejected with a browser one. A build path (emulator .apk, simulator .zip) or an uploaded app id, APP followed by six or more digits. On-device package ids are not accepted.
no_reset: optional. Keep the app’s existing state between runs instead of resetting it.
device_name / os_version: optional per-test defaults for the device, in the vocabulary of kane-cli devices list --target <kind> (local) or kane-cli devices list --target <kind> --remote (grid catalog). The run flags --device-name / --os-version override them. A name needs a version.

The nested form, target: {platform, app}, is not accepted. The parser refuses it and spells out the flat shape above.

Mobile tests run with kane-cli testmd run and in batch with testrun. Locally that needs macOS Apple Silicon, and setup is covered in Mobile Testing. With testrun run --remote, the suite runs on a grid emulator or simulator from any machine.

Root-only vs root-or-per-step

Root only — Chrome settings (target, chrome_profile, cdp_endpoint, ws_endpoint, headless), the mobile keys (app, no_reset), url, mode, tags, and on_lock_conflict. These apply to the whole run; setting them on an individual step is a parse error.
Root or per-step — everything else can appear in the per-step yaml block to override the frontmatter for that step only.

Title and Steps

After the frontmatter, add a # H1 title for the test. This is purely decorative. Kane CLI ignores everything before the first ## heading. Each ## H2 heading marks a test step. The heading text is a label for your reference; the agent reads the step body underneath it. Write the body as either plain English prose describing what the agent should do, or a single @import <path> line to pull in a reusable helper file. Do not mix prose and @import in the same step.

Per-Step Config Overrides

You can override frontmatter settings for individual steps by adding a yaml fenced block immediately under the step heading:

## Submit the form
```yaml
timeout: 90
optional: true
```
Click submit and verify the confirmation banner.

Setting optional: true tells Kane CLI that a failure on this step should not fail the overall test. The run continues to the next step.

`optional`

optional: true marks a step as soft-failing. If the step fails:

The overall run does not halt.
Subsequent steps still execute.
The step is reported as failed in Result.md with an (optional) suffix.

Default is false, in which case any failure stops the run and marks the remaining steps as skipped. optional is allowed on every prose step. On an @import step it is allowed only at the top level of a test file — see Composing tests with @import.

Step body shapes

The body of a step (everything after the optional yaml block) must be exactly one of:

A prose objective — one or more lines of natural language describing what the agent should do. A step body follows the same rules as any objective, so end it in a check of the result and phrase actions as goals — see Writing objectives.
An @import — a single line of the form @import <path> and nothing else.

Mixing prose and @import in the same body is a parse error.

## OK — prose
Click the cart icon and verify two items are listed.

## OK — import
@import ./helpers/login.md

## NOT OK — both
Click the cart icon.
@import ./helpers/login.md

Replay and Cascade Rule

This is the most important concept in testmd.

Replay

After the first run, each step replays from its cached recording with no AI agent, no LLM cost, and much faster execution. A step replays only if all of these hold:

A recording for that step exists
The step’s prose is unchanged since the recording
The step’s yaml block is unchanged
No earlier step invalidated it

Cascade

Editing step N re-authors step N and every step after it. Each step starts where the previous step left off (URL, login state, open tabs). When step 3 changes, step 4 cannot safely replay against state that no longer exists.

A one-line tweak at the top of a 20-step test re-authors all 20 steps on the next run. To minimize re-authoring, edit only the steps you need to change.

Useful commands:

Action	How
Re-record one step	Edit only that step (steps after it cascade automatically)
Force full re-authoring	Use `--author` flag for one run
Wipe cache entirely	Run `rm -rf output-<stem>/`

Reusing Flows with `@import`

Extract repeating flows (login, setup, cookie banner dismissal) into helper files:

## Sign in
@import ./helpers/login.md

Rules:

Helper filename must not end in _test.md. Only _test.md files are valid entry points
Path resolves relative to the importing file, not the shell’s working directory
The step body must be exactly @import <path> with no mixed prose or extra lines
The step’s yaml block may contain only optional
optional: true on @import is allowed only at the root file level, not on nested imports
Variables and context propagate into helpers automatically

Editing a helper re-authors that step in every test that imports it, plus everything after the import in those tests. The same cascade rule applies.

Full detail: Composing tests with @import.

Variables

Variables let you parameterise objectives with reusable values and secrets. Reference a variable inside any prose objective using its placeholder syntax (default {{name}}):

## Sign in
Open the login page and sign in as {{tester_email}} with password {{tester_password}}.

Defining variables in a test.md

Variables can be set in three places inside a _test.md file, in order of increasing specificity:

---
variables:
  tester_email:
    value: "alice@example.com"
  tester_password:
    value: "s3cret-pa55"
    secret: true
---

## Switch to the staging tenant
```yaml
variables:
  tenant: "staging-eu"
```
Open https://{{tenant}}.example.com and verify the login page loads.

Shorthand name: "value" works too — tester_email: "alice@example.com" is equivalent to tester_email: { value: "alice@example.com" }.

Field	Required	Type	Default	Description
`value`	yes	string	—	The variable’s value. Entries without `value` are rejected.
`secret`	no	boolean	`false`	When `true`, the value is masked in logs and routed to the secrets store.
`syntax`	no	string	`{{<name>}}`	Custom placeholder syntax.

Secrets

Secrets are variables with secret: true. They are masked in displayed output, redacted in Result.md, and routed to TestmuAI’s secrets store instead of being synced as plain Test Manager variables.

variables:
  api_key:
    value: "sk-live-abc123"
    secret: true

Context

Context files are plain Markdown text that the agent reads alongside your objective. Use them for standing instructions — coding conventions, accounts to use, sites to avoid.

---
global_context: "~/work/team-conventions.md"
local_context: ".testmuai/project-notes.md"
---

You can pass either a file path (resolved at runtime) or inline Markdown text. See Variables and context for the full discovery rules.

Recording a Test from a Live Session

Run an ad-hoc objective with the --name flag to save it as a replayable test file:

kane-cli run "Search for noise-cancelling headphones on amazon.com" --name amazon-search

On exit, Kane CLI writes the test file to .testmuai/tests/amazon-search_test.md. Move that file into your repo and re-run it with testmd run. The recorded file is a regular _test.md — it is parsed and executed exactly like a hand-written one, and you can edit it freely.

Without --name, ad-hoc runs are ephemeral and nothing is written to disk.

Commands

Command	Purpose
`kane-cli testmd run <path> --agent`	Run a test file
`kane-cli testmd list`	List `*_test.md` files under the current directory
`kane-cli testmd status <path>`	Show Test Manager identity and local sync state
`kane-cli testmd export <path>`	Regenerate code export from existing recordings (no browser launch)
`kane-cli testmd delete <path>`	Delete the test and its `output-<stem>/` cache locally. Does NOT delete from Test Manager

Full detail: Running test.md files.

Flags for `testmd run`

All kane-cli run flags apply (--agent, --headless, --max-steps, --timeout, --variables, etc.), plus these additional flags:

Flag	Default	Description
`--name <name>`	none	Persist the run under this name. Must match `[a-zA-Z0-9_-]+`
`--on-lock-conflict <mode>`	none	Behavior when another user holds the test’s edit lock: `readonly` (replay-only, no upload), `fail` (exit 2), or `wait` (block until released)
`--retry`	off	On replay failure, restart with a shrinking replay window
`--retry-count <n>`	`3`	Max retry restarts before falling back to full re-author
`--author`	off	Force authoring every step, skipping replay entirely
`--code-language <lang>`	`python`	Code export language: `python` or `javascript`

Flag values win over frontmatter for all settings except variables. The file owns variables. You can add new keys via flags but cannot override file-defined ones.

Output Directory

After a run, Kane CLI creates an output directory next to the test file:

amazon_test.md
output-amazon/
  Result.md                      # Human-readable run report
  .internal/                     # Cached recordings, do not edit
  playwright-python-code/        # Only if code_export is enabled

output-<stem>/ is commit-safe. Commit it to git so teammates and CI replay the same recordings. For tests using @import, helper recordings land next to the helper file in helper-output-<helper>-<root>-<step>/ directories. These are also commit-safe.

Result.md

After each run, Result.md is generated with:

Section	Contents
Frontmatter	`status`, `started`, `duration_s`, `session_id`
Step results	One entry per step: `✓ passed`, `✗ failed`, or `⏭ skipped` (suffixed `(optional)` for soft-failing optional steps)
Import failure paths	For `@import` steps that failed, the path to the failing sub-step inside the helper

To check whether a test passed or where it failed, read Result.md instead of re-running the test.

CI/CD Usage

kane-cli testmd run ./tests/checkout_test.md \
  --agent \
  --headless \
  --on-lock-conflict wait \
  --retry

Flag	Purpose
`--agent`	Structured NDJSON to stdout
`--headless`	No browser window
`--on-lock-conflict wait`	Block instead of failing if a teammate is editing the same test
`--retry`	Automatically recover from transient replay failures

Exit Codes

Code	Meaning
0	✅ Passed
1	❌ Failed (test ran but did not pass)
2	⚠️ Error (auth, setup, parse error, or `--on-lock-conflict fail`)
3	⏱️ Timeout, cancelled, or `--on-lock-conflict wait` timed out

Common Parse Errors

Parse errors abort before any browser launch with exit code 2:

Error Message	Fix
`frontmatter is missing closing '---'`	Add the trailing `---`
`invalid YAML in frontmatter`	Validate the YAML block
step-config fenced ```yaml block is not closed	A per-step `yaml` block is missing its closing ```.
`invalid YAML in step config: <details>`	Per-step YAML is malformed.
`step body must be exactly one of prose / @import`	Split into two steps
`step config 'optional' must be boolean: got <type>`	`optional` was set to a non-boolean.
`variable '<k>' must be a string or { value: ... } object`	A variable entry is the wrong shape.
`step config on @import may only contain 'optional'`	Remove other keys from the yaml block
`cannot @import a test file`	Imports may only reference helpers (not files ending in `_test.md`)
`cyclic reference`	Restructure helpers to break the circular dependency
`chrome config is global-only`	Move Chrome key to root frontmatter
`'<key>' is run-level and cannot be set per-step`	Move `mode` / `on_lock_conflict` to root frontmatter
`tags must be a non-empty list of non-empty strings`	`tags:` is present but empty, or one of its entries is empty.
`unknown config key`	Remove or fix the key
`auth/identity keys are CLI-only`	Pass `username` / `access_key` as CLI flags, not in frontmatter

Example: Full Test with Imports

tests/checkout_test.md:

---
mode: testing
headless: true
variables:
  username:
    value: "testuser@example.com"
  password:
    value: "s3cret!"
    secret: true
---

# Checkout flow

## Login
@import ./helpers/login.md

## Add item to cart
Go to the products page, search for "wireless headphones", and click "Add to Cart" on the first result.

## Verify cart
Go to the cart page. Assert the cart contains 1 item. Store the total price as 'cart_total'.

## Complete checkout
Click "Proceed to Checkout", fill in shipping details, and assert the order confirmation page loads.

tests/helpers/login.md:

# Login helper

## Open the login page
Go to https://app.example.com/login.

## Sign in
Enter {{username}} in the email field and {{password}} in the password field, then click "Sign In". Assert the dashboard loads.

Run:

kane-cli testmd run tests/checkout_test.md --agent

Next Steps

Writing Objectives: Learn how to write effective natural language objectives
Variables & Context: Parameterize tests with variables, secrets, and context files
Agent Mode: Parse structured NDJSON output from Kane CLI
CI/CD Integration: Add Kane CLI to your pipeline
CLI Reference: Full flag and command reference
Troubleshooting: Debug common issues

​Quick Start

​When to Use testmd vs run

​File Format

​YAML Frontmatter

​Mobile Target

​Root-only vs root-or-per-step

​Title and Steps

​Per-Step Config Overrides

​optional

​Step body shapes

​Replay and Cascade Rule

​Replay

​Cascade

​Reusing Flows with @import

​Variables

​Defining variables in a test.md

​Secrets

​Context

​Recording a Test from a Live Session

​Commands

​Flags for testmd run

​Output Directory

​Result.md

​CI/CD Usage

​Exit Codes

​Common Parse Errors

​Example: Full Test with Imports

​Next Steps

Quick Start

When to Use testmd vs run

File Format

YAML Frontmatter

Mobile Target

Root-only vs root-or-per-step

Title and Steps

Per-Step Config Overrides

`optional`

Step body shapes

Replay and Cascade Rule

Replay

Cascade

Reusing Flows with `@import`

Variables

Defining variables in a test.md

Secrets

Context

Recording a Test from a Live Session

Commands

Flags for `testmd run`

Output Directory

Result.md

CI/CD Usage

Exit Codes

Common Parse Errors

Example: Full Test with Imports

Next Steps