How to Write Kane CLI Tests in Markdown

testmd lets you write browser tests as Markdown files (_test.md) and commit them to your repo. On the first run, the AI agent authors each step and saves a recording. On every subsequent run, each step replays from cache with no LLM cost and much faster execution. Commit the test file and its recordings to git so teammates and CI can re-run the same tests without re-authoring. A _test.md file is a kane-cli test you can commit to git. It captures the same kind of natural-language objective you would type into kane-cli run, but in a structured Markdown file that:

Lives in your repository alongside the code it tests.
Reads as a human-friendly description of what the test does.
Splits a flow into named steps you can edit independently.
Caches each step on first run, so a passing test replays in seconds the next time you run it.
Composes: one test can @import another file to reuse a flow such as login.

This page covers the file format end-to-end. To run a test once you have written one, see Running test.md files. To reuse flows across tests, see Composing tests with @import.

Quick Start

Step 1: Create a test file. The filename must end in _test.md:

---
mode: testing
max_steps: 30
---

# Amazon search

## Open Amazon
Open https://www.amazon.com.

## Search for headphones
Type "wireless headphones" into the search box and submit.
Verify at least one product result is visible.

Step 2: Run the test:

kane-cli testmd run amazon_test.md --agent

The agent launches Chrome, works through each step in order, writes a result file to output-amazon/Result.md, and exits 0 if everything passed. On the first run, the agent authors each step and caches the recording. On every later run, the steps replay from cache instantly.

Always combine --agent with --headless in CI/CD environments to avoid display server errors.

When to Use testmd vs run

kane-cli run is one-shot. It runs an objective, uploads results, and exits. It is ideal for quick, one-off verifications like checking if a page loads correctly or extracting a value from a live site. kane-cli testmd run is for tests you want to persist. Use it when you are building a login flow smoke test, a regression suite, or any test you plan to re-run across builds. The test file lives in your repo, recordings are cached and committed alongside it, and every subsequent run replays from cache without consuming LLM credits. Teammates and CI pick up the same recordings and replay them identically. If you run an ad-hoc objective with kane-cli run and later decide you want to keep it, use the --name flag to save it as a _test.md file (see Recording a Test from a Live Session below).

File Format

A _test.md file has four parts in order:

---
key: value         # frontmatter (YAML)
---

# Optional title

## Step 1 heading
Plain prose objective describing what the agent should do.

## Step 2 heading
```yaml             # optional per-step config block
timeout: 60
optional: true
```
Another prose objective.

## Step 3 heading
@import ./helpers/login.md

Frontmatter: run-wide settings inside --- ... --- at the top of the file.
## Step heading: every step starts with an H2. Anything before the first ## is silently ignored, so the optional # Title is decorative.
Optional yaml block: fenced ```yaml ... ``` immediately under the step heading, overriding the frontmatter for that step only.
Step body: either a prose objective or a single @import <path> line. Not both.

File name

Files that end in _test.md are tests. They are valid entry points for kane-cli testmd run. Files that do not end in _test.md are helpers and can only be reached through @import from a test. See Composing tests with @import.

YAML Frontmatter

Frontmatter is YAML inside --- fences at the top of the file. Every key is optional:

---
mode: testing
max_steps: 30
headless: true
---

Supported keys:

Key	Type	Scope	Description
`mode`	`"action"` \| `"testing"`	root only	How the agent reacts to auth walls, blocked pages, and error pages. `testing` (default) pushes through so negative-test assertions can run; `action` halts so you can intervene. See Run mode.
`url`	string	root only	Start URL for the test’s first step. Bare domains are normalized to `https://`. Overridden by the `--url` flag; falls back to the configured `default_url`. See Default start URL.
`tags`	list or string	root only	Labels for selecting tests in batch runs. Accepts a YAML list, a bracketed inline list (`tags: [smoke, checkout]`), or a bare comma string (`tags: smoke, checkout`). Tags are trimmed, lowercased, and de-duplicated. Shown by `kane-cli testmd list` and the interactive picker; selected with `testrun --tags`; recorded in the run’s evidence pack.
`max_steps`	integer	root + per-step	Maximum number of agent reasoning steps for the run or step. The CLI default is `50`. If neither the step config nor the CLI supplies a value, the engine uses `30`.
`timeout`	integer (seconds)	root + per-step	Time limit for each step; the step is stopped when it runs out. No default.
`global_context`	string (Markdown text) or file path	root + per-step	Standing instructions inlined into the agent’s context. See Variables & Context.
`local_context`	string or file path	root + per-step	Project-scoped guidance. Same shape as `global_context`.
`variables`	object	root + per-step	Named values usable as `{{name}}` in objectives. See Variables.
`session_context`	`{ prior_runs: [...] }`	root + per-step	Context from earlier runs, loaded for the agent before the run starts.
`target`	`"chrome"` \| `"cdp"` \| `"ws"` \| `"emulator"` \| `"simulator"` \| `"android"` \| `"ios"`	root only	Where the test runs: a browser transport (`chrome`, the default, `cdp`, or `ws`), a virtual Android or iOS device (`emulator` or `simulator`, locally on macOS Apple Silicon or on the cloud grid with `testrun run --remote`), or a real Android or iOS device in the TestMu AI cloud (`android` or `ios`, from any machine). See Mobile Target.
`app`	string	root only	Mobile only. The app under test: a build path (emulator `.apk`, simulator `.zip`, android `.apk` or `.aab`, ios `.ipa`) or an uploaded `APP…` id. Required with a mobile target, rejected with a browser target. On the grid, a real run uploads simulator `.zip` builds from your machine; an existing `APP…` id also works. A dry run does not upload anything.
`no_reset`	boolean	root only	Mobile only. Keep the app’s existing state between runs instead of resetting it.
`device_name`	string	root only	Mobile only. The default device for this test, as `kane-cli devices list --target <kind>` prints it (add `--remote` for the grid catalog). Overridden by `--device-name`.
`os_version`	string	root only	Mobile only. The device’s OS version (`14`, `17.5`). Overridden by `--os-version`. Required alongside `device_name`.
`chrome_profile`	string	root only	Named Chrome profile under `~/.testmuai/kaneai/chrome-profiles/`.
`cdp_endpoint`	string	root only	Reuse an external Chrome over CDP.
`ws_endpoint`	string	root only	LambdaTest / Playwright WebSocket endpoint.
`headless`	boolean	root only	Launch Chrome without a visible window.
`code_export`	boolean	root + per-step	Generate Playwright code after the run. See Code export.
`code_language`	`"python"` \| `"javascript"`	root + per-step	Code-export target language.
`on_lock_conflict`	`"readonly"` \| `"fail"` \| `"wait"`	root only	Policy when another user holds the lock on this test in Test Manager. See Lock conflicts.

Keys not in this table are rejected with unknown config key: <key> at parse time. Authentication is set via CLI flags or your active profile, never in frontmatter.

Mobile Target

target: also accepts four mobile values: emulator for a virtual Android device, simulator for a virtual iOS device, and android or ios for a real device in the TestMu AI cloud. The app under test is its own root key:

---
target: emulator             # emulator | simulator (virtual) · android | ios (real device)
app: ./builds/app-debug.apk  # or an APP… id from `kane-cli apps list --target emulator`
device_name: Pixel 7         # optional default device (with os_version)
os_version: "14"             # optional; `--device-name` / `--os-version` override both
no_reset: false              # optional
---

target: emulator runs on an Android emulator, simulator runs on an iOS simulator, and android or ios runs on a real device. You do not set the platform separately; the target determines it.
app: the app under test. It is required with a mobile target and rejected with a browser target. Use a build path (emulator .apk, simulator .zip, android .apk or .aab, ios .ipa) or an uploaded app id, which is APP followed by six or more digits. On-device package ids are not accepted.
no_reset: optional. Keeps the app’s existing state between runs instead of resetting it. Refused with android or ios, because a real-device session always starts with a fresh install.
device_name / os_version: optional default device for this test. Use the names exactly as kane-cli devices list --target <kind> prints them (local), or kane-cli devices list --target <kind> --remote (grid catalog, or real devices). The run flags --device-name / --os-version override them. On an emulator or simulator, if you set a name, also set a version. On a real device, you can set either one alone, and a name can be a pattern (Pixel.*). If you set neither, the hub (the TestMu AI service that assigns real devices) picks any free device.

The nested form, target: {platform, app}, is not accepted. The parser rejects it and shows the flat form above in its error message.

Run mobile tests with kane-cli testmd run, or in batches with testrun. Running emulator and simulator tests locally requires macOS on Apple Silicon; see Mobile Testing for setup. With testrun run --remote, the suite runs on a grid emulator or simulator, and you can start it from any machine. Simulator members accept a local .zip build or an APP… id in app:. Real-device tests run from any machine with no setup beyond kane-cli login. See Real Devices.

Root-only vs root-or-per-step

Root only: Chrome settings (target, chrome_profile, cdp_endpoint, ws_endpoint, headless), the mobile keys (app, no_reset), url, mode, tags, and on_lock_conflict. These apply to the whole run; setting them on an individual step is a parse error.
Root or per-step: everything else can appear in the per-step yaml block to override the frontmatter for that step only.

Title and Steps

After the frontmatter, add a # H1 title for the test. This is purely decorative. Kane CLI ignores everything before the first ## heading. Each ## H2 heading marks a test step. The heading is purely a label for humans and for the result file; the agent reads the step body underneath it, not the heading. Write the body as either plain English prose describing what the agent should do, or a single @import <path> line to pull in a reusable helper file. Do not mix prose and @import in the same step.

Per-Step Config Overrides

You can override frontmatter settings for individual steps by adding a yaml fenced block immediately under the step heading. The block must be closed with a matching ```.

## Submit the form
```yaml
timeout: 90
optional: true
```
Click submit and verify the confirmation banner.

Allowed keys: any root-or-per-step key from the frontmatter table, plus optional (see below). Chrome keys, mode, and on_lock_conflict are rejected with a clear error if you try to set them here. Setting optional: true tells Kane CLI that a failure on this step should not fail the overall test. The run continues to the next step.

`optional`

optional: true marks a step as soft-failing. If the step fails:

The overall run does not halt.
Subsequent steps still execute.
The step is reported as failed in Result.md with an (optional) suffix.

Default is false, in which case any failure stops the run and marks the remaining steps as skipped. optional is allowed on every prose step. On an @import step it is allowed only at the top level of a test file. See Composing tests with @import.

Step body shapes

The body of a step (everything after the optional yaml block) must be exactly one of:

A prose objective: one or more lines of natural language describing what the agent should do. A step body follows the same rules as any objective, so end it in a check of the result and phrase actions as goals. See Writing objectives.
An @import: a single line of the form @import <path> and nothing else.

Mixing prose and @import in the same body is a parse error.

## OK: prose
Click the cart icon and verify two items are listed.

## OK: import
@import ./helpers/login.md

## NOT OK: both
Click the cart icon.
@import ./helpers/login.md

Replay and Cascade Rule

This is the most important concept in testmd.

Replay

After the first run, each step replays from its cached recording, with no AI agent, no LLM cost, and much faster execution. An authorable step replays only when all of the following are true. (Steps marked replay-only always replay, so these conditions do not apply to them.)

A recording for that step exists
The step’s prose is unchanged since the recording
The step’s yaml block is unchanged
No earlier step invalidated it

Cascade

Editing step N re-authors step N and every authorable step after it. Steps marked replay-only still replay. This is because each step starts where the previous step left off (URL, login state, open tabs). If step 3 changes, the state step 4 depends on may be different, so step 4 cannot safely replay from cache.

A one-line tweak at the top of a 20-step test re-authors all authorable steps on the next run. To minimize re-authoring, edit only the steps you need to change.

Useful commands:

Action	How
Re-record one step	Edit only that step (steps after it cascade automatically)
Force full re-authoring	Pass `--author` for one run. Every authorable step is authored again; steps marked replay-only still replay
Recover from a replay failure	Adaptive healing is on by default; pass `--no-adaptive-heal` to turn it off

Keep the output recordings for steps marked replay-only. Deleting them does not make those steps author again. See Replay policy and completion.

For replay-only heading markers and balanced structured control flow (@if/@end-if, @while/@end-while), see Replay policy and completion.

Reusing Flows with `@import`

Extract repeating flows (login, setup, cookie banner dismissal) into helper files:

## Sign in
@import ./helpers/login.md

Rules:

Helper filename must not end in _test.md. Only _test.md files are valid entry points
Path resolves relative to the importing file, not the shell’s working directory
The step body must be exactly @import <path> with no mixed prose or extra lines
The step’s yaml block may contain only optional
optional: true on @import is allowed only at the root file level, not on nested imports
Variables and context propagate into helpers automatically

Editing a helper re-authors that step in every test that imports it, plus everything after the import in those tests. The same cascade rule applies.

Full detail: Composing tests with @import.

Variables

Variables let you parameterise objectives with reusable values and secrets. Reference a variable inside any prose objective using its placeholder syntax (default {{name}}):

## Sign in
Open the login page and sign in as {{tester_email}} with password {{tester_password}}.

Defining variables in a test.md

Variables can be set in three places inside a _test.md file, in order of increasing specificity:

---
variables:
  tester_email:
    value: "alice@example.com"
  tester_password:
    value: "s3cret-pa55"
    secret: true
---

## Switch to the staging tenant
```yaml
variables:
  tenant: "staging-eu"
```
Open https://{{tenant}}.example.com and verify the login page loads.

Shorthand name: "value" works too: tester_email: "alice@example.com" is equivalent to tester_email: { value: "alice@example.com" }.

Field	Required	Type	Default	Description
`value`	yes	string	none	The variable’s value. Entries without `value` are rejected.
`secret`	no	boolean	`false`	When `true`, the value is masked in logs and routed to the secrets store.
`syntax`	no	string	`{{<name>}}`	Custom placeholder syntax.

Where else variables can come from

Variables defined inline in a _test.md are merged with variables loaded from JSON files and command-line flags. The full pipeline is documented in Variables & Context. One difference matters for test.md authors:

Inside a _test.md file, variables declared in frontmatter or in a step’s yaml block win over --variables and --variables-file for the same key.

Every other flag overrides what is in the file; variables are the exception. The reasoning: the test author’s choice of test data stays close to the test. You can still add new keys from the CLI that the file does not define.

Secrets

Secrets are variables with secret: true. They are masked in displayed output, redacted in Result.md, and routed to TestMu AI’s secrets store instead of being synced as plain Test Manager variables.

variables:
  api_key:
    value: "sk-live-abc123"
    secret: true

Context

Context files are plain Markdown text that the agent reads alongside your objective. Use them for standing instructions: coding conventions, accounts to use, sites to avoid.

---
global_context: "~/work/team-conventions.md"
local_context: ".testmuai/project-notes.md"
---

You can pass either a file path (resolved at runtime) or inline Markdown text. See Variables and context for the full discovery rules.

Recording a Test from a Live Session

You can author a test.md by hand, or record one from an interactive kane-cli run session. Run an ad-hoc objective with the --name flag to save it as a replayable test file:

kane-cli run "Search for noise-cancelling headphones on amazon.com" --name amazon-search

On exit, Kane CLI writes the session to <cwd>/.testmuai/tests/amazon-search_test.md. Move that file into your repo and re-run it with testmd run, as any other test. The recorded file is a regular _test.md. It is parsed and executed exactly like a hand-written one, and you can edit it freely.

Without --name, ad-hoc runs are ephemeral and nothing is written to disk.

Commands

Command	Purpose
`kane-cli testmd run <path> --agent`	Run a test file
`kane-cli testmd list`	List `*_test.md` files under the current directory
`kane-cli testmd status <path>`	Show Test Manager identity and local sync state
`kane-cli testmd export <path>`	Regenerate code export from existing recordings (no browser launch)
`kane-cli testmd delete <path>`	Delete the test and its `output-<stem>/` cache locally. Does NOT delete from Test Manager

Full detail: Running test.md files.

Flags for `testmd run`

All kane-cli run flags apply (--agent, --headless, --max-steps, --timeout, --variables, etc.), plus these additional flags:

Flag	Default	Description
`--name <name>`	none	Persist the run under this name. Must match `[a-zA-Z0-9_-]+`
`--on-lock-conflict <mode>`	none	What to do when another user holds the test’s edit lock: `readonly` (replay only, no upload), `fail` (exit 2), or `wait` (wait until the lock is released)
`--no-adaptive-heal`	healing enabled	Turn off adaptive healing after a replay failure
`--author`	off	Re-author all authorable steps; steps marked replay-only still replay
`--code-language <lang>`	`python`	Code export language: `python` or `javascript`

Flag values win over frontmatter for all settings except variables. The file owns variables. You can add new keys via flags but cannot override file-defined ones.

Full flag reference: Running test.md files.

Output Directory

After a run, Kane CLI creates an output directory next to the test file:

amazon_test.md
output-amazon/
  Result.md                      # Human-readable run report
  .internal/                     # Cached recordings, do not edit
  playwright-python-code/        # Only if code_export is enabled

output-<stem>/ is commit-safe. Commit it to git so teammates and CI replay the same recordings. For tests using @import, helper recordings land next to the helper file in helper-output-<helper>-<root>-<step>/ directories. These are also commit-safe.

Result.md

After each run, Result.md is generated with:

Section	Contents
Frontmatter	`status`, `started`, `duration_s`, `session_id`
Step results	One entry per step: `✓ passed`, `✗ failed`, or `⏭ skipped` (suffixed `(optional)` for soft-failing optional steps)
Import failure paths	For `@import` steps that failed, the path to the failing sub-step inside the helper

To check whether a test passed or where it failed, read Result.md instead of re-running the test.

CI/CD Usage

kane-cli testmd run ./tests/checkout_test.md \
  --agent \
  --headless \
  --on-lock-conflict wait

Flag	Purpose
`--agent`	Structured NDJSON to stdout
`--headless`	No browser window
`--on-lock-conflict wait`	Block instead of failing if a teammate is editing the same test

By default, adaptive healing recovers from replay failures. Pass --no-adaptive-heal to turn it off.

Exit Codes

Code	Meaning
0	✅ Passed
1	❌ Failed (test ran but did not pass)
2	⚠️ Error (auth, setup, parse error, or `--on-lock-conflict fail`)
3	⏱️ Timeout, cancelled, or `--on-lock-conflict wait` timed out

Worked Example

A complete test that signs in, searches, adds to cart, and verifies the badge, using a variable for the search term and a per-step timeout.

---
mode: testing
max_steps: 35
headless: false
variables:
  search_term:
    value: "wireless headphones"
---

# Amazon: add to cart

## Open Amazon
Open https://www.amazon.com.

## Search
Type "{{search_term}}" into the search box and submit.
Verify at least one product result is visible.

## Open the first result
```yaml
timeout: 60
```
Click the first non-sponsored product card. Verify the product detail page loads.

## Add to cart
Click "Add to Cart" on the product page.
Verify the cart icon shows a count of 1 or higher.

The first run of this file authors every step: the agent works out the page and records each step. Later runs replay each passed step from cache in a fraction of the time. If you edit a step’s objective, that step and the steps after it in the same file are authored again on the next run. See Replay vs author for the rules.

Common Parse Errors

The parser catches these mistakes early. A parse error stops the run with exit code 2 before any browser launch, auth call, or upload:

Error Message	Fix
`frontmatter is missing closing '---'`	Add the trailing `---`
`invalid YAML in frontmatter: <details>`	Validate the YAML block
step-config fenced ```yaml block is not closed	A per-step `yaml` block is missing its closing ```.
`invalid YAML in step config: <details>`	Per-step YAML is malformed.
`step body must be exactly one of prose / @import`	Split into two steps
`step config 'optional' must be boolean: got <type>`	`optional` was set to a non-boolean.
`variable '<k>' must be a string or { value: ... } object`	A variable entry is the wrong shape.
`step config on @import may only contain 'optional': got <key>`	Remove other keys from the yaml block
`cannot @import a test file`	Imports may only reference helpers (not files ending in `_test.md`)
`cyclic reference`	Restructure helpers to break the circular dependency
`chrome config is global-only: <key>`	A Chrome-related key was set on an individual step. Move it to root frontmatter
`'<key>' is run-level and cannot be set per-step`	`mode`, `tags`, or `on_lock_conflict` was set on an individual step. Move it to root frontmatter
`tags must be a non-empty list of non-empty strings`	`tags:` is present but empty, or one of its entries is empty.
`unknown config key: <key>`	A frontmatter or per-step key is not recognised. Remove or fix the key
`auth/identity keys are CLI-only: <key>`	`username`, `access_key`, or another auth key appeared in frontmatter. Pass them as CLI flags

Example: Full Test with Imports

tests/checkout_test.md:

---
mode: testing
headless: true
variables:
  username:
    value: "testuser@example.com"
  password:
    value: "s3cret!"
    secret: true
---

# Checkout flow

## Login
@import ./helpers/login.md

## Add item to cart
Go to the products page, search for "wireless headphones", and click "Add to Cart" on the first result.

## Verify cart
Go to the cart page. Assert the cart contains 1 item. Store the total price as 'cart_total'.

## Complete checkout
Click "Proceed to Checkout", fill in shipping details, and assert the order confirmation page loads.

tests/helpers/login.md:

# Login helper

## Open the login page
Go to https://app.example.com/login.

## Sign in
Enter {{username}} in the email field and {{password}} in the password field, then click "Sign In". Assert the dashboard loads.

Run:

kane-cli testmd run tests/checkout_test.md --agent

Next Steps

Running test.md files: the kane-cli testmd run command, flags, the replay model, and what the output directory contains
Batch runs with testrun: run many tests at once, selected by path or tags:
Composing tests with @import: break a long test into reusable helpers
Writing Objectives: Learn how to write effective natural language objectives
Variables & Context: Parameterize tests with variables, secrets, and context files
Agent Mode: Parse structured NDJSON output from Kane CLI
CI/CD Integration: Add Kane CLI to your pipeline
CLI Reference: Full flag and command reference
Troubleshooting: Debug common issues

​Quick Start

​When to Use testmd vs run

​File Format

​File name

​YAML Frontmatter

​Mobile Target

​Root-only vs root-or-per-step

​Title and Steps

​Per-Step Config Overrides

​optional

​Step body shapes

​Replay and Cascade Rule

​Replay

​Cascade

​Reusing Flows with @import

​Variables

​Defining variables in a test.md

​Where else variables can come from

​Secrets

​Context

​Recording a Test from a Live Session

​Commands

​Flags for testmd run

​Output Directory

​Result.md

​CI/CD Usage

​Exit Codes

​Worked Example

​Common Parse Errors

​Example: Full Test with Imports

​Next Steps

Quick Start

When to Use testmd vs run

File Format

File name

YAML Frontmatter

Mobile Target

Root-only vs root-or-per-step

Title and Steps

Per-Step Config Overrides

`optional`

Step body shapes

Replay and Cascade Rule

Replay

Cascade

Reusing Flows with `@import`

Variables

Defining variables in a test.md

Where else variables can come from

Secrets

Context

Recording a Test from a Live Session

Commands

Flags for `testmd run`

Output Directory

Result.md

CI/CD Usage

Exit Codes

Worked Example

Common Parse Errors

Example: Full Test with Imports

Next Steps