Cut Test Execution Time for High-Velocity Releases

Why Is Test Execution the Release Bottleneck?

A suite that ran nightly could take an hour and nobody minded. The same suite gating every merge is a tax paid on every commit, by every engineer, several times a day.

There is a published number for how long that gate should take. DORA's continuous integration guidance puts the test suite at a few minutes with an upper limit of about ten, and its test automation guidance repeats the bound, adding that flaky tests should not be tolerated.

Most pipelines sit outside it. In CircleCI's 2026 State of Software Delivery, the median workflow across its platform runs 2.2 minutes while the mean runs 9.9 minutes, and the typical team takes 72 minutes to recover from a failed workflow, a 13% increase year over year.

A mean sitting that far above the median says a long tail of slow pipelines owns the pain, rather than describing the typical run. A recovery time longer than most suites take to execute says a fast suite that fails opaquely still costs the afternoon.

Speed and diagnosis are therefore one problem rather than two. A suite that halves its runtime but leaves engineers reading logs to work out what broke has moved the delay rather than removed it.

For the vocabulary behind the numbers, the test execution guide covers phases and entry criteria as a discipline.

Where Does the Wall Clock Actually Go?

Teams buy concurrency to fix a number they have not decomposed. A run is not one duration, it is six, and only one of them responds to more machines.

That asymmetry explains most disappointing parallelisation results.

• Setup cost multiplies as you widen - provisioning and installation are per-machine, so a job split twelve ways pays them twelve times. Wall-clock time still falls because they happen concurrently, but total compute rises, which is the honest trade behind every speed number on this page.
• There is a floor you cannot cross - once execution is spread thin enough, the remaining duration is upload plus provisioning plus install plus discovery. Adding machines past that point buys nothing, and the first fix is caching rather than concurrency.

Measure the segments before buying more parallelism. If installation is the largest bar, the useful change is a cache key, not a bigger plan.

Four Ways Teams Make a Suite Finish Faster

These four stack rather than compete. Most teams reach for them in this order, and the failure mode is skipping to the last one while the first is still misconfigured.

# One process per physical core, tests grouped so a file stays on one worker
pytest -n auto --dist loadfile

# Rebalance at runtime instead of trusting the initial split
pytest -n 8 --dist worksteal

# Each command runs on its own machine, with reporter: 'blob' set in
# playwright.config so the shard emits a blob-report/ directory
npx playwright test --shard=1/4
npx playwright test --shard=2/4
npx playwright test --shard=3/4
npx playwright test --shard=4/4

# In a final job, once every shard's blob-report/ has been collected
# into one all-blob-reports/ folder, merge them into a single report
npx playwright merge-reports --reporter html ./all-blob-reports

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      max-parallel: 4
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

A Measured Serial Baseline

Before optimising anything, measure the thing you are optimising. I built a small Playwright suite against the Selenium Playground and ran it on one worker to get a serial number to reason from.

The suite is 12 spec files holding 24 tests on Playwright 1.47.2. Each file performs one real interaction, then walks four playground pages end to end, which is deliberately closer to a real journey test than a single assertion.

$ npx playwright test

Running 24 tests using 1 worker

  ok 1 01-playground-index.spec.js:5:3 › Playground index › playground index lists demo pages (1.2s)
  ok 2 01-playground-index.spec.js:11:3 › Playground index › walks four playground pages end to end (7.0s)
  ok 3 02-simple-form.spec.js:5:3 › Simple form › single input message is echoed back (2.2s)
  ok 4 02-simple-form.spec.js:14:3 › Simple form › walks four playground pages end to end (6.4s)
  ok 5 03-checkbox.spec.js:5:3 › Checkbox › check all selects every option checkbox (2.2s)
  ok 6 03-checkbox.spec.js:15:3 › Checkbox › walks four playground pages end to end (6.3s)
  ok 7 04-dropdown.spec.js:5:3 › Dropdown › day and multi-select values can be set (2.1s)
  ok 8 04-dropdown.spec.js:13:3 › Dropdown › walks four playground pages end to end (6.3s)
  ok 9 05-input-form.spec.js:5:3 › Input form › submitting an empty form is blocked (621ms)
  ok 10 05-input-form.spec.js:11:3 › Input form › walks four playground pages end to end (6.9s)
  ok 11 06-table-search.spec.js:5:3 › Table search › sorting a column reorders the first row (4.0s)
  ok 12 06-table-search.spec.js:18:3 › Table search › walks four playground pages end to end (6.6s)
  ...
  ok 23 12-download-progress.spec.js:5:3 › Download progress › download progress demo page renders (599ms)
  ok 24 12-download-progress.spec.js:11:3 › Download progress › walks four playground pages end to end (7.3s)

  24 passed (1.6m)

The suite takes 1.6 minutes serially. More usefully, the per-file spread runs from roughly 1.6 seconds to about 9.5 seconds, which is exactly the imbalance that count-based sharding cannot see.

Scale that shape rather than that size. A suite of 600 files with the same distribution sits near 80 minutes serially, and the same uneven tail decides how much of the split is wasted.

Findings from writing the suite, both of which cost real time

• A duplicate DOM id broke a strict locator - the simple form demo page carries both an input and a div with id user-message, so addressing the id alone is ambiguous under strict-mode locators and the fill silently targeted the wrong node. Scoping the selector to input#user-message fixed it.
• The widgets hydrate after load - navigating with domcontentloaded found empty containers, and four tests failed on selectors that exist only after hydration. Waiting for networkidle turned all four green without touching the assertions.

Both are ordinary authoring bugs rather than platform problems, and both would have arrived as intermittent failures under parallelism. That is the pattern worth internalising: parallelising a suite with latent timing assumptions converts hidden bugs into flaky ones.

How HyperExecute Splits the Same Suite

Distribution strategy is the largest single lever on wall-clock time, and HyperExecute exposes three. The choice is decided by where your parallelism comes from: the test files, the environments, or both.

The Config That Runs It

Adoption is one file in the project root plus one binary. Below is a complete Auto Split configuration for the Playwright suite measured above, with every key that touches duration called out afterwards.

---
version: "0.1"

runson: linux

autosplit: true
concurrency: 12

globalTimeout: 90
testSuiteTimeout: 90
testSuiteStep: 90

pre:
  - npm ci
  - npx playwright install --with-deps chromium

cacheKey: '{{ checksum "package-lock.json" }}'
cacheDirectories:
  - node_modules

testDiscovery:
  type: raw
  mode: dynamic
  command: find tests -name '*.spec.js' | sort

testRunnerCommand: npx playwright test $test

retryOnFailure: true
maxRetries: 2

failFast:
  maxNumberOfTests: 3

report: true
partialReports:
  location: results
  type: xml
  frameworkName: junit

jobLabel: [regression, playwright, linux]

The CLI reads that file and orchestrates the run. It authenticates from environment variables rather than flags in your shell history.

# Linux
curl -O https://downloads.lambdatest.com/hyperexecute/linux/hyperexecute
chmod +x hyperexecute

export LT_USERNAME="your_username"
export LT_ACCESS_KEY="your_access_key"

./hyperexecute --config hyperexecute.yaml

Confirm the binary version before writing config, because schema rules changed between YAML versions and error messages reference the version you are on. The build used while preparing this page reported the following.

$ ./hyperexecute --version
HyperExecute version 0.2.355

Version 0.2 is not a minor revision of 0.1. It removes testDiscovery, testRunnerCommand, and Matrix mode entirely and requires a framework directive, so a 0.1 example handed to a 0.2 user does not work in either direction. The YAML parameters reference is the place to confirm a key before shipping it.

• cacheKey and cacheDirectories - keyed on the lockfile checksum, this turns the per-machine install from a download into a restore, which is the single largest saving on a wide split. Smart caching covers what is safe to cache.
• concurrency - the machine count for Auto Split, and required for it. Sizing guidance lives in finding the correct concurrency, and the platform surfaces a recommendation once it has seen your usage.
• failFast - aborts the job after a threshold of consecutive failures so a uniformly broken build stops burning machines. The counter resets on a pass, which keeps it targeted at broken builds rather than flaky ones.
• globalTimeout - ranges from 1 to 150 minutes and defaults to 90, so a job that genuinely needs longer has to be split into several jobs rather than raised further.
• uploadArtefacts - the artefact key appears in the docs under both the British and American spellings, and the CLI accepts either. Its path takes a list, not a string, and passing a bare string is the mistake that actually fails validation.
• What --validate does not catch - the CLI type-checks the keys it knows and reports missing required fields, but a key it does not recognise is ignored silently. A typo in a key name therefore passes validation and then does nothing at runtime.

Keeping a Fast Suite Trustworthy

A fast suite nobody believes is worse than a slow one, because it fails without changing anyone's behaviour. Speed work has to carry reliability work alongside it.

Flakiness is not scattered noise, which changes how you should attack it. An analysis of 10,000 test suite runs across 24 Java projects published as Systemic Flakiness found that 75% of flaky tests belong to a cluster of co-occurring failures, with a mean cluster size of 13.5 tests.

That clustering is good news operationally: a dozen flaky tests usually share one root cause, so the fix is one shared fixture rather than thirteen patches. The flaky test guide covers the diagnosis patterns.

Scoped retries deserve the emphasis. Listing error patterns in retryOptions.errorRegexps means a driver timeout is retried while an assertion failure fails immediately, which is the honest answer to the objection that retries hide defects. Check the scope before relying on it: the documentation lists this flag as working with Cypress, CDP, and Selenium tests, so a Playwright suite gets the blunt retryOnFailure and maxRetries pair instead.

On the diagnosis side, failures are classified into product bug, test automation bug, environment issue, and no action required, and Test Intelligence carries the root cause analysis that turns a failed task into a named cause with remediation steps. Given a 72-minute typical recovery time, cutting triage is worth as much as cutting execution.

Wiring It Into the Release Pipeline

One suite rarely serves both a pull request and a nightly regression well. Splitting the config by intent is what keeps the pull-request gate inside the ten-minute bound while still running everything overnight.

The trigger is the same binary in every case, which is what makes this a configuration split rather than three pipelines. Job priority labels of high, medium, or low let a release-critical suite clear the queue ahead of routine runs when concurrency is contended.

The CI/CD integration documentation covers wiring the trigger into Jenkins, GitHub Actions, GitLab, CircleCI, and the rest of the toolchain. For the broader pipeline design around it, the guide to CI/CD pipeline best practices for test automation speed covers the stages either side of execution.

Coverage breadth is the other reason the nightly stage differs. TestMu AI's Automation Cloud covers 3000+ browser, OS, and device combinations, which is what a matrix stage draws on when it fans out across environments.

When Is Cloud Orchestration the Wrong Answer?

Orchestration is not free, and there are suites it will slow down. These are the cases where the honest recommendation is to keep what you have.

• The suite already finishes inside the bound - if a runner you already pay for completes the suite in under ten minutes, adding payload upload, provisioning, and queue time makes the gate slower, not faster.
• Setup dominates execution - a suite whose tests take seconds but whose installation takes minutes will barely move under any split. Fix the cache first, then reconsider concurrency.
• Concurrency is plan-bounded - setting concurrency above what the plan allows does not buy more parallelism, so the plan has to be sized to the suite rather than the YAML.
• The job needs more than 150 minutes - globalTimeout tops out at 150 minutes, so a genuinely longer job has to be divided into several jobs.
• Some features are not YAML keys - auto-healing is a WebDriver capability rather than a YAML setting, test muting is a dashboard preference, and flaky test detection currently supports Selenium-based tests with a Playwright reporter integration rather than every framework.
• Windows-only optimisations - the smartGrid browser-state caching optimisation is available on Windows only, so a Linux-only estate does not get it.

Auto-healing carries its own boundary, and it is worth stating plainly. It cannot recover from WebDriver initialisation errors, and aggressive healing can mask a genuine regression where an element really did disappear, so it belongs on cosmetic locator drift rather than on functional coverage.

And a suite with shared state does not become correct by being distributed. Parallelising tests that collide over the same fixture or account converts a latent ordering bug into an intermittent failure, which is the pattern the two authoring bugs in the baseline section illustrate.

Measured Results From Production Suites

TestMu AI publishes an upper bound of up to 70% faster execution than traditional cloud on its test orchestration page, and it is worth reading as a ceiling rather than a per-suite guarantee. The customer numbers below are more useful because they name the before state.

Read the Boomi figure as a combined result rather than a platform benchmark. Its case study attributes the drop to modularising tests and creating data dynamically alongside the move to cloud execution, and quotes Hrishi Potdar, Quality Engineering Architect at Boomi, saying the team tripled its tests and now executes in less than 2 hours.

The Transavia split is the more instructive shape. Its published breakdown separates the main test suite from the secondary one, and the gap between the two is exactly what the segment analysis predicts: the suite carrying more parallelisable execution time gained far more than the one where setup dominated.

Conclusion

Time one serial run and write down the six segments before changing anything. That single measurement tells you whether your problem is execution, in which case concurrency helps, or setup, in which case a cache key helps and concurrency does not.

Then put the existing suite behind one hyperexecute.yaml file with Auto Split and caching on, and compare against that baseline rather than against a percentage from a vendor page. The getting started documentation covers account setup and the first job, and the HyperExecute orchestration cloud is where the strategy, concurrency, and reporting all live.