How to gate a pull request with Kane CLI in CI

Pass test variables to Kane CLI tests in CI

Tests that use {{name}} variables, such as a login test, need their values on the runner, and testrun has no --variables-file flag. Add a step before the Kane CLI action that writes them from repository secrets into .testmuai/variables/, which every run loads. Without it, the run types {{username}} literally and the test fails.

- name: Write Kane CLI test variables
  env:
    TEST_USERNAME: ${{ secrets.TEST_USERNAME }}
    TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
  run: |
    mkdir -p .testmuai/variables
    jq -n --arg u "$TEST_USERNAME" --arg p "$TEST_PASSWORD" \
      '{username:{value:$u},password:{value:$p,secret:true}}' > .testmuai/variables/ci.json

Gate a pull request on any CI with one testrun command

Outside GitHub Actions, install Node 18 or later, Chrome and Kane CLI on the runner, start your app, then run the suite with credentials from your secret store:

npm install -g @testmuai/kane-cli
kane-cli testrun run --tags smoke --parallel 4 --headless --agent \
  --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"

--headless is required, because CI runners have no display. --agent forces the NDJSON stream. Most CI runners select it anyway because stdin is not a terminal, but on a runner that attaches a terminal, testrun without it opens an evidence table that never exits.

Archive .testmuai/evidence/*.evidence and each test's output-<stem>/Result.md as build artifacts so reviewers can open the result, and bound the suite with your CI job's timeout.

Kane CLI testrun exit codes a CI job can gate on

A local run's NDJSON stream ends with testrun_done, whose overall_status is passed, failed or cancelled; a --remote run adds remote_done after it. Gate on the exit code, because a run refused before it starts can end without either event.

Run the committed tests before a coding agent pushes

When a coding agent should check its own work before it opens a pull request, it runs the committed tests, not new objectives:

kane-cli testrun run --tags smoke --headless --agent

Exit 0: continue with the commit or pull request. Exit 1: stop, read the failing test's output-<stem>/Result.md for the step, and report it to the user with the screenshot from the evidence pack in .testmuai/evidence/ instead of pushing. Exit 2: nothing ran; report it as "did not start" with the fix (sign-in, Chrome or an invalid suite plan) and do not push. Exit 3: the suite was cancelled; rerun it before you push.

Run a Kane CLI test suite on the TestMu AI cloud grid

To run the suite on the cloud grid rather than on the CI runner, install the plugin in the job (each CI runner starts clean) and preview the execution plan first. --dry-run resolves the execution plan without creating a job.

kane-cli plugin install remote-execution
kane-cli testrun run --tags smoke --remote --dry-run --agent \
  --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"
kane-cli testrun run --tags smoke --remote --parallel 4 --agent \
  --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"

--remote needs a TestMu AI subscription that includes HyperExecute. The working directory is sent as the job and respects .gitignore, so keep the output-<stem>/ folders out of .gitignore. Use it with tests that open a deployed or preview URL, not an app that runs on the CI runner itself.

Kane CLI docs for CI gating

• Official GitHub Action: inputs, outputs, evidence and the pull request comment.
• CI/CD recipes: headless runs, credentials and evidence artifacts.
• Suites with testrun: selection, parallel workers and exit codes.
• Remote execution: what the cloud grid needs.