When a coding agent should save a run as a test
When a check is worth repeating: a sign-in flow, a checkout, a feature you just shipped or a bug you just fixed. A committed test runs again on every change without anyone rewriting it.
Steps to save, replay and commit a Kane CLI test
- 1
Record a named run with
--nameSign in first (
kane-cli login --oauth, or ask the developer to runkane-cli login). The name takes letters, digits,_and-. Credentials come from a variables file the developer fills in, such as.testmuai/variables/local.jsonshaped like{"username":{"value":""},"password":{"value":"","secret":true}}. Kane CLI loads that folder on every run.bashkane-cli run "Go to http://localhost:3000, log in with {{username}} and {{password}}, then verify the dashboard heading is visible" --agent --headless --name login - 2
Move the _test.md file and its recordings into your repo
Kane CLI writes the test and its
output-login/folder next to each other under.testmuai/tests/. Move them together: a step replays only when its recording is there, so a test moved without its folder is re-authored on its next run.bashmkdir -p tests && mv .testmuai/tests/login_test.md .testmuai/tests/output-login tests/ - 3
Tag a saved test so suites can select it
Add
tagsto the file's frontmatter, for exampletags: [smoke, auth]. Tags are lowercased, and an empty list is a parse error. - 4
Replay a saved test with testmd run
Under
--agent, the stream ends withtest_md_done(overall_status,duration_s,session_id). Each step also emits its ownrun_end, which closes that step, not the file. Use the exit code too:0passed,1a step failed,2setup or parse error,3timeout or cancelled.bashkane-cli testmd run tests/login_test.md --agent --headless - 5
Commit the _test.md file with its recordings
Commit
tests/login_test.mdtogether withtests/output-login/. Tests that use@importalso writehelper-output-folders next to the helper, one for each place a test imports it, so commit those too. Do not commit.testmuai/evidence/(run artifacts) or.testmuai/variables/(credentials): supply those values on each machine, and in CI write them from secrets (see Gate a pull request in CI).
What a Kane CLI test file looks like
A _test.md file has YAML frontmatter, an optional # Title, then one ## heading per step with a plain-English body. The file name must end in _test.md; any other .md file is a helper that tests reach with @import. Read this format to tag or edit a recorded test, and get new tests from --name runs or kane-cli design tests rather than writing them from scratch.
---
mode: testing
tags: [smoke, checkout]
---
# Checkout
## Open the cart
Open http://localhost:3000/cart.
## Pay
Pay with card {{card_number}}, then verify the confirmation page shows an order number.Write test data into the step text or put it in a variables file: {{card_number}} comes from a file in .testmuai/variables/. A run with no terminal cannot stop to ask for a value.
How Kane CLI replays and re-authors a saved test
- A step replays from its recording, with no LLM cost, only if the recording exists, its text and
yamlblock are unchanged, and no earlier step was re-authored. - If you edit step N, steps N onward are re-authored on the next run, so edit only the step that changed.
- When a replayed step no longer matches the page, Kane CLI re-authors from that step onward, with up to three attempts, and then re-authors the whole test. Pass
--no-adaptive-healto make a replay miss fail the run instead. - Tests on real devices (
target: androidorios) have no adaptive heal.
Share one flow across tests with @import
Put a shared flow, such as signing in, in a helper file whose name does not end in _test.md, and make @import ./helpers/login.md the whole body of a step. The path resolves from the importing file.
Edit the helper once and every test that imports it uses the new version. On the next run, the edited helper step, the helper steps after it, and every step after the @import in each importing test are re-authored.
Run several saved tests as one suite
For two or more tests, use kane-cli testrun run instead of looping over testmd run: you get one summary, one exit code and one evidence pack.
kane-cli testrun run --tags smoke --parallel 4 --headless --agentKeep --agent on the command, or redirect stdin with < /dev/null in bash. When stdin is a terminal and neither is there, testrun opens an evidence table that waits for q and never exits on its own. The stream ends with testrun_done. To make the suite block merges, see Gate a pull request in CI.
Kane CLI docs for saved tests
- test.md files: the file format, frontmatter and step syntax.
- Running test.md files: replay rules, recordings,
Result.mdand healing. - Composition with @import: helpers, variables and what re-authors after an edit.
Questions about saving Kane CLI runs as tests
What happens to a saved test when the UI changes?
When a replayed step no longer matches the page, Kane CLI re-authors from that step onward, with up to three attempts, and then re-authors the whole test. Pass --no-adaptive-heal if a replay miss should fail the run instead.
Does replaying a saved test use Kane CLI's AI?
Not for unchanged steps: they replay from the recording with no LLM cost. Kane CLI's AI authors a step again when its text or yaml block changed (and every step after it), when it has no recording, when the run uses --author, or when healing re-authors a failed replay.
Should a coding agent write _test.md files by hand from a spec?
No. Kane CLI designs tests from requirements. kane-cli context ingest <file> --mode agent extracts use cases, the developer approves them with kane-cli context review, kane-cli design tests --use-case <id> --mode agent writes one test per scenario, kane-cli testmd run authors each one once, and kane-cli cover shows what the evidence proves against what the requirements owe. On these commands, exit 3 means paused, not failed.
