How to Debug a Failed Kane CLI Run

When a run fails, start with its evidence pack. Everything is in one place, and every log line and screenshot is tied to the step that produced it.

The four steps

Open the pack. Accept the post-run offer, or run kane-cli evidence serve <pack> and open the viewer URL.
Go to the failed step. The run overview marks it. Its failure record shows the error message and the state of the page when the step failed.
Check the step’s console and network activity. Logs are split by step, so you see exactly what the browser logged and requested while that step ran. A 4xx or 5xx response or a JavaScript error in these logs usually explains the failure.
Look at the annotated screenshot. It highlights the element the agent acted on, so failures such as “clicked the wrong thing” or “element was not there” are easy to spot.

Reading a failure record

A failed or broken step normally has its own failure.yaml. It holds the error, the page state at the time of failure, and references into the console and network logs. The pack root has a failure.yaml index that collects all of them, so you can see every failure in the run without opening each step. Every record must show what went wrong: an error.message, an expected and an actual value, or both.

`failed` or `broken`

The verdict tells you where to look first.

Verdict	What it means	Where to look
`failed`	The test’s check (its oracle) ran and the product behaved incorrectly.	The assertion and the page state. This is a likely defect.
`broken`	The check could not run because of an environment, infrastructure, or test fault.	The network log and the run log. The product may be fine.

If you treat both verdicts the same, a flaky environment can end up filed as a product bug. Keeping them separate prevents that.

Working from the command line

A sealed pack is a zip file, so you can answer quick questions without the viewer:

# list everything in the pack
unzip -l <execution_id>.evidence

# print the run manifest
unzip -p <execution_id>.evidence run.yaml

# print one test's result
unzip -p <execution_id>.evidence tests/<test-id>/result.yaml

When the pack itself looks wrong

If a pack does not open, run kane-cli evidence validate. An unsealed pack, for example from a run that was force-killed, gets the structure checks only and can still report as valid. A truncated pack cannot be read at all. In both cases, the run’s pack is also in the session directory:

~/.testmuai/kaneai/sessions/<session-id>/evidence/

Next steps

Pack structure: where each artifact lives.
Validating packs: check a suspect pack.

​The four steps

​Reading a failure record

​failed or broken

​Working from the command line

​When the pack itself looks wrong

​Next steps