Close the Loop: Turn Production Telemetry and Incident Logs Into Tests With Kane CLI

Scenario 1: A Single Failed Page Load

The simplest case. Your error tracker reports a 500 on a specific route.

Signal:

[error] 500 on GET /pricing | trace_id=a1b2c3

Distilled objective:

go to https://app.example.com/pricing,
assert the page loads without a 500 or error banner,
assert the pricing table is visible

Run:

kane-cli run "go to https://app.example.com/pricing, assert the pricing table is visible, assert no error banner" --agent --headless

If it fails, you have reproduced the outage against a real browser, with a video and a network capture, in seconds. If it passes, the failure was transient or state-dependent, and you know to look at load or data conditions rather than the route itself.

Scenario 2: A Broken Multi-Step Flow

Now the log points at a user journey, not a single page. A checkout error is the classic example.

Signal:

{ "url": "/checkout", "action": "place order", "expected": "confirmation",
  "observed": "spinner hangs, 500 in network tab" }

Distilled objective:

go to https://app.example.com/checkout,
add a Visa test card ending 4242,
click Place Order,
assert the order confirmation page loads,
assert no error message is visible

Run with test data injected:

kane-cli run "go to https://app.example.com/checkout, fill card with '{{card_number}}', click Place Order, assert order confirmation loads" \
  --variables-file ./secrets.json --agent --headless

Because Kane CLI resolves elements by intent rather than brittle selectors, this same objective keeps reproducing the bug after the checkout UI is redesigned, which matters once the test lives in your suite for months.

Scenario 3: A Conditional Bug for a User Segment

Harder. The incident only affects users in a specific state, say, accounts with an expired trial. The log carries that context, and the objective has to set it up.

Signal:

{ "url": "/dashboard", "segment": "trial_expired",
  "action": "click Upgrade", "observed": "Upgrade button disabled, no path forward" }

Distilled objective, split into stages:

Stage 1: log in as a trial-expired test account,
go to the dashboard,
assert the trial-expired banner is visible

Stage 2: click Upgrade,
assert the upgrade plan selection page opens,
assert the Upgrade button is not disabled

Split conditional setups into stages. Kane CLI holds browser state across objectives in a session, so stage 1 establishes the condition and stage 2 tests the behavior. This is how you reproduce "only happens when" bugs that a single flat script cannot express.

Scenario 4: An SSO-Gated Internal App

Enterprise incidents often land behind a login wall, on internal tools where the failing flow is three menus deep, and the account is SSO-protected with MFA. A flat script chokes on the SAML round-trip; this workflow does not.

Signal:

{ "app": "servicenow", "url": "/incident/new", "auth": "Okta SSO + TOTP",
  "action": "submit incident with attachment",
  "observed": "Submit spins, attachment never uploads" }

Distilled objective, auth first:

Stage 1: log in via Okta SSO using the test account,
complete TOTP from the provided secret,
assert the ServiceNow home loads

Stage 2: open a new incident,
attach the sample file,
click Submit,
assert the incident number is displayed,
assert no upload error is shown

Two enterprise-specific points. Kane CLI holds the authenticated session across stages, so you authenticate once and test the deep flow after. And MFA is not a blocker: KaneAI supports TOTP natively, so a test account protected by an authenticator app can be driven with the TOTP secret your IT team issues for that account. Internal apps built on nested menus and shadow DOM resolve by intent rather than by selectors that break every release.

Scenario 5: A Regulated Banking Flow With Data Verification

In regulated software, reproducing the UI failure is only half the job. You also have to confirm the data behind it, and do it without touching real customer records.

Signal:

{ "app": "retail-banking", "url": "/transfer", "segment": "joint_account",
  "action": "transfer between own accounts",
  "observed": "UI shows success, balance does not update" }

Distilled objective with a data assertion:

log in as a joint-account test user,
go to Transfers,
transfer 100 from Checking to Savings,
assert the confirmation shows 'Transfer complete',
store the displayed Savings balance as 'ui_balance'

The UI half reproduces the "looks successful but isn't" bug directly, the most dangerous class in finance because the happy path lies. For the data half, KaneAI can run a database step to query the backing store and assert the persisted balance matches the value shown on screen, catching the exact gap between what the interface claims and what actually committed.

Every value here is synthetic: a test joint account, a test transfer, a staging database. Regulated flows are precisely where you must never replay real balances, real PII, or real credentials from a production log.

Scenario 6: The Full Agentic Loop

The complex case ties it together: an incident lands, a coding agent translates it, Kane CLI verifies it, and a confirmed reproduction is committed, all without a human writing the test.

Agent-side pseudocode:

on new incident:
  ctx = extract(incident)          # url, action, expected, observed
  objective = to_objective(ctx)    # LLM turns intent into a Kane CLI objective
  result = run kane-cli(objective, "--agent --headless")
  if result.status == "failed":    # reproduced
    open_pr(objective, result.share_link)
  else:
    comment(incident, "not reproducible from this path; needs state or data")

The agent parses Kane CLI's NDJSON output, reads the terminal run_end verdict and exit code, and branches on it. A reproduced bug becomes a pull request with the objective and a link to the run. A non-reproduction becomes a note back on the incident itself, a useful signal: the bug depends on something the log did not capture.

In CI, the promoted test gates every future build:

- name: Reproduction suite
  run: kane-cli run-suite ./tests --agent --headless

The incident that reached one customer becomes a check that blocks it for all of them. The same idea, wired into a pull request, is how teams keep Kane CLI CI/CD-ready without a second framework.

The same Kane CLI syntax runs on your laptop and in CI, so a reproduction you confirm locally becomes a pipeline gate with no rewrite. You can see the worked examples in the Kane CLI repository.