How to Perform Regression Testing From the Command Line

The Regression Testing Loop as Commands

Running everything on every change is still the default. TestMu AI's future of quality assurance survey found that 73.8% of teams run all automation tests every time, which it ties directly to longer developer feedback times. A loop of four commands, each doing one job, cuts that feedback time without shrinking the suite.

# 1. Select       2. Scope to the change       3. Shard       4. Rerun failures
npx playwright test --grep "@regression" --only-changed=main --shard=1/4
npx playwright test --last-failed

pytest -m regression -n auto          # pytest-xdist for parallel workers
pytest --lf

npx jest -t regression --changedSince=main --shard=1/4
npx jest --onlyFailures

mvn test -Dgroups=regression -Dsurefire.rerunFailingTestsCount=1

The framework examples use a six-test Playwright suite that runs against TestMu AI's Selenium Playground on the TestMu AI cloud grid; the Kane CLI section uses a demo Shopify storefront. If you need a refresher on what belongs in the suite, start with regression testing and how it differs in sanity testing vs regression testing.

How to Select the Regression Suite With Tags

A tag records intent in the test file, and the command line decides what runs. In Playwright the tag sits in the test details object, so one test can belong to both the fast smoke subset and the full regression suite.

// tests/forms.spec.ts - tags live in the test, selection happens on the command line
test('single checkbox toggles on', { tag: ['@regression', '@smoke'] }, async ({ page }) => {
  await page.goto('checkbox-demo/');
  const box = page.getByRole('checkbox', { name: 'Click on check box' });
  await box.check();
  await expect(box).toBeChecked();
});

# Playwright
npx playwright test --grep "@regression" --list          # preview, run nothing
npx playwright test --grep "@smoke"
npx playwright test --grep "@regression" --grep-invert "@slow"

# pytest (register markers in pytest.ini to avoid warnings)
pytest -m regression
pytest -m "regression and not slow"

# Jest (matches the full test name, including describe blocks)
npx jest -t "regression"

# Maven + JUnit 5 tags through Surefire
mvn test -Dgroups=regression
mvn test -Dgroups=regression -DexcludedGroups=slow

Always preview a new filter before trusting it in CI. --list collects the matching tests and runs nothing, which catches a typo in a tag before a pipeline passes with zero tests:

$ npx playwright test --grep "@smoke" --list
Listing tests:
  forms.spec.ts:3:5 › single input echoes the message
  forms.spec.ts:22:5 › single checkbox toggles on
Total: 2 tests in 1 file

• Quote the tag - "@regression" in quotes survives every shell. Unquoted, PowerShell treats @ as its own syntax and the filter arrives empty.
• Combine, don't duplicate - --grep-invert or pytest's "and not slow" carve a subset out of the regression suite without adding another tag to maintain.
• Java suites - Surefire passes -Dgroups straight to JUnit 5 tags or TestNG groups. For the underlying runner, see run JUnit from command line.
• Smoke subset after a deploy - gating a release on the @smoke tag, with separate exit codes for rollback and retry, is covered in smoke testing command line.

How to Run Only the Tests a Change Affects

Scope pull request runs to the files a change touches. The test CLI reference describes --only-changed as running only test files changed between HEAD and a ref, with uncommitted changes as the default. Jest's CLI goes further with --changedSince and --findRelatedTests, which follow imports from changed source files to the tests that cover them.

# Playwright: test files changed between HEAD and main (uncommitted changes by default)
npx playwright test --only-changed=main

# Jest: tests related to files changed since main, or to named source files
npx jest --changedSince=main
npx jest --findRelatedTests src/cart/total.ts

On a feature branch that edits only the dropdown test, Playwright runs that one file, three of the six regression tests:

$ git diff --stat main
 tests/widgets.spec.ts | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

$ npx playwright test --only-changed=main
Running 3 tests using 1 worker
  ok 1 tests/widgets.spec.ts:3:5 › radio button reports the chosen value @regression (1.9s)
  ok 2 tests/widgets.spec.ts:13:5 › select dropdown shows the selected day @regression (996ms)
  ok 3 tests/widgets.spec.ts:19:5 › table lists the Tokyo office @regression (1.1s)
  3 passed (20.7s)

File-level selection misses a shared helper or a backend change that no test file imports, so treat it as the pull request gate and keep a full nightly run as the safety net. Deciding what a change can reach is its own discipline, covered in impact analysis in testing.

How to Shard and Parallelize a Regression Run

Sharding splits one suite across several machines, and workers split each shard across processes. In CI each shard is usually one matrix job; GitHub's workflow syntax caps a matrix at 256 jobs per workflow run, far more shards than most suites need.

$ npx playwright test --grep "@regression" --shard=1/2 --workers 3
Running 3 tests using 1 worker, shard 1 of 2
  ok 1 tests/forms.spec.ts:3:5 › single input echoes the message @regression @smoke (1.4s)
  ok 2 tests/forms.spec.ts:12:5 › two inputs return their sum @regression (1.4s)
  ok 3 tests/forms.spec.ts:22:5 › single checkbox toggles on @regression @smoke (834ms)
  3 passed (17.0s)

$ npx playwright test --grep "@regression" --shard=2/2 --workers 3
Running 3 tests using 1 worker, shard 2 of 2
  ...
  3 passed (21.9s)

Both shards passed, but each ran on one worker despite --workers 3. Playwright keeps the tests in a file on a single worker unless the run is fully parallel, and each shard here received one file. Add --fully-parallel (or fullyParallel: true in the config) when the tests do not share state.

# Playwright: split the suite, and let tests inside one file run in parallel too
npx playwright test --grep "@regression" --shard=2/4 --fully-parallel

# pytest-xdist: one process per physical core, whole files kept on one worker
pytest -m regression -n auto --dist loadfile

# Jest
npx jest --shard=2/4

• Balance by file count - shards split by test files, so one oversized spec file becomes the slowest shard. Break it up before adding more shards.
• pytest-xdist distribution - --dist loadfile keeps every test in a file on the same worker, which protects module-level fixtures that are not safe to share.

How to Rerun Failures and Catch Flaky Tests

Flaky tests are a measurable cost. The same TestMu AI survey found teams spend 7.8% of their time fixing flaky tests, and that 58% of teams see more than 1% of their test runs come back flaky. On the command line, rerun only what failed, then measure whether a failure repeats.

When a full run fails one test, fix it and pass --last-failed to rerun only that test instead of all six:

$ npx playwright test --grep "@regression"
  ok 2 tests/widgets.spec.ts:3:5 › radio button reports the chosen value @regression (1.6s)
  ...
  1) tests/forms.spec.ts:3:5 › single input echoes the message @regression @smoke
  1 failed
  5 passed (39.3s)

$ npx playwright test --last-failed
Running 1 test using 1 worker
  ok 1 tests/forms.spec.ts:3:5 › single input echoes the message @regression @smoke (1.4s)
  1 passed (14.8s)

To check whether a failure is flaky, repeat the test with one retry allowed. Playwright's test retries guide defines a flaky test as one that fails first and passes on retry, and the run shows the trap: one flaky result, and still exit code 0.

$ npx playwright test --grep radio --repeat-each 8 --retries 1
  ok 1 tests/widgets.spec.ts:3:5 › radio button reports the chosen value @regression (1.6s)
  ...
  ok 9 tests/widgets.spec.ts:3:5 › radio button reports the chosen value @regression (retry #1) (1.6s)
  1 flaky
  7 passed (55.2s)
$ echo $?
0

Here the cause is timing: the page's button handler attaches after the script loads, so an early click does nothing. Clicking the label instead still fails four of eight repeats, and waiting for network idle passes but is discouraged in Playwright's docs. Retrying the whole interaction inside expect().toPass() passes eight of eight:

test('radio button reports the chosen value', { tag: '@regression' }, async ({ page }) => {
  await page.goto('radiobutton-demo/');
  // The button responds only after the page's script loads, so retry the interaction until it does.
  await expect(async () => {
    await page.getByRole('radio', { name: 'Male' }).first().check();
    await page.getByRole('button', { name: 'Get value', exact: true }).click();
    await expect(page.getByRole('main')).toContainText("'Male' is checked", { timeout: 1_000 });
  }).toPass({ timeout: 15_000 });
});

pytest --lf                                   # only the last failures
pytest --ff                                   # failures first, then everything else
npx jest --onlyFailures
mvn test -Dsurefire.rerunFailingTestsCount=2  # reruns in the same build, reports "Flakes: N"

Maven behaves like Playwright here: its rerun failing tests option reports a test that passes on rerun as a flake and still marks the build successful. Count the flakes in CI, or they stay invisible.

Gating the Regression Suite in CI

The workflow below combines the earlier steps: changed tests on each pull request, the full suite nightly, four shards, one retry, and a hard failure on flaky results. Per the workflow syntax reference, fail-fast defaults to true in a matrix, which cancels in-progress shards the moment one fails; set it to false so every shard reports.

name: regression
on:
  pull_request:
  schedule:
    - cron: '0 2 * * *'             # full suite every night

jobs:
  regression:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false              # the default (true) cancels the other shards on one failure
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0            # --only-changed needs the base branch history
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - name: Changed tests (pull requests)
        if: github.event_name == 'pull_request'
        env:
          LT_USERNAME: ${{ secrets.LT_USERNAME }}
          LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
        run: >
          npx playwright test --grep "@regression"
          --only-changed=origin/${{ github.base_ref }}
          --shard=${{ matrix.shard }}/4 --retries=1 --fail-on-flaky-tests
      - name: Full regression suite (nightly)
        if: github.event_name == 'schedule'
        env:
          LT_USERNAME: ${{ secrets.LT_USERNAME }}
          LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
        run: npx playwright test --grep "@regression" --shard=${{ matrix.shard }}/4 --retries=1

• Exit code as the gate - Playwright, pytest, Jest, and Maven all exit non-zero on a failed test, so a required status check blocks the merge with no report parsing.
• Flaky results block on PRs only - --fail-on-flaky-tests on pull requests forces a fix; the nightly run records flakes without blocking anything.
• Same pattern for mobile - the matrix, secrets, and exit-code gate carry over to device runs, as shown in mobile app testing command line.
• Full user journeys - starting the app, saving a login, and keeping traces for a checkout flow are covered in end-to-end testing command line.

How to Run Regression Tests With Kane CLI

If the regression suite is written as plain-English tests instead of code, Kane CLI runs it with the same select, parallelize, and fail-fast controls as the frameworks above. Each test is a _test.md file: YAML front matter carries the tags, and every ## heading is one step with its objective in plain English.

The first run authors each step with the agent. Every later run replays the recording, and that rerun of unchanged tests against a changed application is the regression check.

The walkthrough uses the Shopify storefront from the theme update smoke use case, a change that can break buying without anyone noticing. A theme is published in admin, the storefront is checked for the new theme, and the buyer path is walked to checkout.

Two sibling tests from the same storefront, checkout extension render and discount code application, share the regression tag and make the folder a suite. The commands and output shown are from Kane CLI 0.8.18 on Windows 11; the same commands run unchanged on macOS and Linux.

npm install -g @testmuai/kane-cli
kane-cli --version
kane-cli login

0.8.18

kane-regression\
  tests\shopify\
    theme-update-smoke_test.md         # tags: [regression, smoke]
    checkout-extension-render_test.md  # tags: [regression]
    discount-code-application_test.md  # tags: [regression]

---
mode: testing
tags: [regression, smoke]
variables:
  store_url: "https://my-testing-repo-main.vercel.app/shopify-clone-app"
---

# Theme update smoke

## Confirm the live theme
Open {{store_url}}/admin?reset=true, click "Themes", and verify the themes list shows "Dawn" marked "Live".

## Publish a different theme
Click "Publish" on "Refresh", then click "Publish theme" in the confirmation dialog, and verify a notice appears reading "Refresh is now the live theme."

## Verify the storefront picked it up
Click "View storefront" and verify the storefront shows "Storefront is running the Refresh theme" and the product grid lists "Heavyweight Cotton Tee".

## Walk through checkout
Click "Add to cart" on "Heavyweight Cotton Tee", click "View cart", click "Checkout", and verify the checkout page renders with a "Payment" section and a "Pay" button.

kane-cli testmd list

kane-cli testrun run --tags regression --dry-run

testrun: testrun 3 paths
  parallel=1  on-failure=continue

  + tests/shopify/theme-update-smoke_test.md
  + tests/shopify/checkout-extension-render_test.md
  + tests/shopify/discount-code-application_test.md

3 test(s) selected - plan valid

Step 4: First Run, Every Step Authored

Run the suite with three parallel workers. On the first run there is no recording yet, so the agent authors each step: it reads the page, decides how to satisfy the objective, and records what it did. This is the slow run.

kane-cli testrun run --tags regression --parallel 3

Each member shows authoring while the agent writes its steps. In this run the summary reads 3 passed and 2 authored: the theme test already had a recording from an earlier run, so it replayed, and the whole suite finished in 5m 48s. Each test gets an output-<name>/ folder next to it with Result.md and the recording in .internal/ that later runs replay, and the suite's sealed evidence pack lands in .testmuai/evidence/.

Step 5: Rerun After the Change, Steps Replay

Push the next theme, or deploy the next build, and rerun a test whose file did not change. Nothing in the test file changed, so every step starts from its recording against the new application state:

kane-cli testmd run tests/shopify/theme-update-smoke_test.md
echo $?        # PowerShell: $LASTEXITCODE   cmd: echo %ERRORLEVEL%

The breakdown reads 4 replay and 0 author, and the exit code is 0. The rerun still took 254 seconds, because the recorded path for step 4 missed on the new page state: adaptive heal re-authored that one step, marked retry-promoted, and the run passed without re-authoring the other three.

Adaptive heal applies when a page changes in a way that breaks the recorded path but not the objective: it re-authors only the failing tail, up to three attempts, before a full re-author. If the change did break something, a step whose assertion no longer holds fails with a Reason: line and its screenshot, and the exit code is 1.

For the whole suite, rerun kane-cli testrun run --tags regression --parallel 3 the same way, and add --on-failure fail-fast to stop dispatching new members after the first failure.

Step 6: Change an Expectation, Only the Tail Re-Authors

When a requirement changes, edit the step text. Kane CLI authors that step again and every step after it, while the steps before it keep replaying, so the suite never replays against stale state. Change step 3 to also verify that the product grid lists "Enamel Camp Mug" and rerun just that file:

kane-cli testmd run tests/shopify/theme-update-smoke_test.md

The breakdown reads 2 replay and 2 author: steps 1 and 2 replayed in 2.52 and 2.2 seconds, while steps 3 and 4 were authored again in 49.1 and 56.2 seconds, for 122 seconds in all. Commit the updated output-theme-update-smoke/ folder with the test file so the next CI run replays the new recording.

kane-cli plugin install remote-execution
kane-cli plugin doctor remote-execution
kane-cli testrun run --tags regression --remote --parallel 3 --dry-run
kane-cli testrun run --tags regression --remote --parallel 3

3 test(s) selected - plan valid

remote testrun · hyper · env prod · dispatching…
Job Link: https://hyperexecute.lambdatest.com/hyperexecute/task?jobId=<job id>
✔ [1]  setup-runtime (19s)
✔ [1]  pre (28s)
✔ [1]  discovery (1s)
✔ [1]  tests/shopify/theme-update-smoke_test.md (21s)
✔ [2]  tests/shopify/checkout-extension-render_test.md (18s)
✔ [3]  tests/shopify/discount-code-application_test.md (16s)
Downloading artifacts...
Final Job Status: Completed

testrun : passed

- name: Regression suite on HyperExecute
  env:
    LT_USERNAME: ${{ secrets.LT_USERNAME }}
    LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
  run: |
    npm install -g @testmuai/kane-cli
    kane-cli plugin install remote-execution
    kane-cli testrun run --tags regression --remote --parallel 3 \
      --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"

Note: Run the tagged regression suite on HyperExecute with kane-cli testrun run --remote and your TestMu AI credentials. Sign up for free.

Troubleshooting Command-Line Regression Runs

Each row pairs a symptom you will see in the output with its usual cause and fix. With --remote, the Job Link in the Kane CLI output opens the HyperExecute job with each member's log.

Conclusion

Start by tagging the tests that guard released behavior and running npx playwright test --grep "@regression" --list to confirm the selection. Then add --only-changed to pull requests, shard the nightly run, and turn on --fail-on-flaky-tests once the suite is stable.

For tests written in plain English, save them as tagged _test.md files and run them on HyperExecute with kane-cli testrun run --remote, where every replay after the first run is the regression check. For deciding which tests earn a place in the suite, see automated regression testing.