How to Detect Flaky Tests?

To detect flaky tests, rerun your tests several times under identical conditions and watch for inconsistent results: a test that passes on some runs and fails on others, with no change to the code or environment, is flaky. You then confirm and prioritize those candidates by tracking pass and fail history across builds, calculating a flake rate per test, and running the suite in random order, in isolation, and in parallel to expose order, state, and timing problems. Test observability and test intelligence platforms automate this by mining historical run data and surfacing the flaky tests for you.

What Makes a Test "Flaky"

A flaky test is one that returns different outcomes on the same code without any deliberate change to the test, the application, or the environment. It passes on one run and fails on the next for non-deterministic reasons such as timing, shared state, test order, network conditions, or differences between environments. Detecting flakiness is specifically about identifying and measuring which tests behave this way. It is a separate step from diagnosing the root cause and repairing the test, and from designing tests so they never become flaky in the first place. This guide focuses entirely on detection.

How to Detect Flaky Tests

The core detection technique is repetition under controlled variation. Run the same tests again and again, change one execution variable at a time, and watch which tests change their result. Each method below targets a different class of flakiness.

• Rerun N times on the same commit: Execute the suite, or just the suspect tests, repeatedly against an unchanged commit, for example 20 to 50 times. Any test that does not produce the same result every time is flaky. Scheduling a dedicated nightly or weekly "flaky-detection" run on the same commit surfaces flakes proactively before they block a release.
• Use retries as a diagnostic signal: A test that fails on its first attempt but passes on an automatic retry is a strong flaky candidate. Enable a small, bounded number of retries purely to gather this signal, not to mask the failure.
• Run in random order: Randomizing the execution order exposes order dependencies and shared state. If a test passes in one order and fails in another, it is relying on side effects from earlier tests.
• Run a test in isolation versus the full suite: If a test passes when run alone but fails inside the complete suite, it is being affected by shared resources, leftover data, or global state set up by other tests.
• Run in parallel: New failures that appear only when tests run concurrently point to race conditions, port or file conflicts, or contention over shared test data.
• Compare results across environments: Run the same tests on different browsers, operating systems, devices, or CI machines. A test that fails in one environment but passes in another is environment-dependent flakiness rather than a real defect.

Framework Flags That Surface Flakiness

Most test runners ship with rerun and repeat options you can use to detect, not just absorb, intermittent failures. The point is to record how often a test recovers on rerun so you can flag it.

• pytest (Python): The pytest-rerunfailures plugin reruns failed tests with --reruns N or the @pytest.mark.flaky(reruns=...) marker, and pytest-repeat stress-runs a single test with --count to surface intermittent failures.
• Cypress: Built-in test retries are enabled through the retries configuration, which records the tests that passed only after a retry.
• Playwright: The retries option in the config file reruns failing tests, and --repeat-each runs each test multiple times to stress for flakiness.
• JUnit (Java): The Flaky Test Handler plugin for Jenkins automatically reruns failed tests to separate flaky tests from consistently failing ones.

Treat all of these as detection instruments. A test that needs a retry to pass is recorded as flaky and queued for investigation, not silently passed off as green.

Measure Flakiness with Pass/Fail History and Flake Rate

A single rerun confirms a hunch; history tells you how bad the problem is and which tests to fix first. Store the pass or fail outcome of every test on every build and compute a flake rate for each test over a recent window:

Flake rate = failed runs / total runs, measured over a sliding window of recent builds (for example, the last 14 days), excluding failures that map to a real code regression.

Any test whose results flip without a matching code change, and whose flake rate sits above your tolerance, is a confirmed flaky test. Many teams treat a flake rate above roughly 1 to 2 percent as unacceptable. A common way to triage by severity:

Use Quarantine Flags and CI Analytics

Once a test is detected as flaky, tag or quarantine it so its noise stops failing the pipeline while you investigate. Most CI tools and test frameworks support marking a test as flaky or moving it into a quarantined set whose results are still recorded but no longer block the build. This keeps the detection signal alive without letting one unreliable test mask genuine failures. CI analytics dashboards add value by visualizing flaky-test trends, correlating new flakiness with the commits that introduced it, and alerting the team when a previously stable test starts flipping.

Auto-Detect with Test Observability and Test Intelligence

Doing all of the above manually across thousands of tests is impractical, which is why test observability and test intelligence platforms automate detection. They mine historical run data across builds, identify tests whose status transitions between passed and failed without a corresponding code change, and rank them by flake rate so you always know your worst offenders.

TestMu AITest Analytics is one example of this approach. It uses a machine learning algorithm that flags flaky tests with a flake icon, draws on WebDriver command logs to see the exact steps each test ran, and detects flakiness from the transition of a test's status across a sequence of runs. It then classifies each flaky test by severity, High, Medium, or Low, based on its flake rate, and lets you tune the grouping, the sliding-window size, and the flake-rate threshold to match your suite. The result is that flaky tests are surfaced automatically instead of being chased down by hand.

Capture Evidence on Failure

Flaky failures are, by definition, hard to reproduce, so detection is only useful if you can study a failure when it happens. Configure your runs to capture rich artifacts on every failure: step-by-step command logs, screenshots at the point of failure, a video recording of the session, and the console and network logs. When an intermittent failure recurs, this evidence lets you see whether the flakiness was driven by a slow element, a timing race, a flaky network call, or an environment difference, all of which point you toward the right fix later.

Detection vs. Fixing vs. Avoiding

Detection, fixing, and prevention are three distinct stages. This guide covers detection: surfacing and measuring which tests are flaky. Once a flaky test is confirmed, you move on to How to Fix Flaky Tests?, and over time you How to Avoid Flaky Tests?. A healthy workflow runs all three in a loop: detect, quarantine, fix, and prevent the same pattern from recurring.

Frequently Asked Questions

How do you know if a test is flaky?

A test is flaky if it produces different results on the same code with no changes to the test, the application, or the environment. The clearest way to confirm this is to rerun the test many times under identical conditions. If it passes on some runs and fails on others, it is flaky rather than genuinely broken.

What is a good way to measure flakiness?

Track a flake rate per test, calculated as the number of failed runs divided by the total number of runs over a recent window of builds. Many teams treat a flake rate above roughly 1 to 2 percent as a problem and flag any test whose failures cannot be tied to a real regression for investigation.

Do test retries help detect flaky tests?

Yes, as a diagnostic signal. A test that fails on its first attempt but passes on an automatic retry is a strong flaky candidate. Tools such as pytest-rerunfailures, Cypress retries, Playwright retries, and the JUnit Flaky Test Handler record these retry outcomes so you can spot which tests recover on rerun. Retries surface flakiness; they do not fix it.

Why run tests in random order to find flakiness?

Running tests in a randomized order exposes order dependencies and shared state. If a test passes in one order but fails in another, it is relying on side effects left by earlier tests, which is a common and hard-to-spot source of flakiness.

How do test observability tools detect flaky tests automatically?

Test observability and test intelligence platforms mine historical run data across builds, detect tests whose status flips between passed and failed without a corresponding code change, and rank them by flake rate. TestMu AI Test Intelligence, for example, flags flaky tests with a flake icon, uses WebDriver command logs, and classifies severity as High, Medium, or Low based on the flake rate.

How is detecting flaky tests different from fixing them?

Detection is about identifying and measuring which tests behave inconsistently, using reruns, flake-rate history, and observability tooling. Fixing is the separate step of finding the root cause of a confirmed flaky test and repairing it. You detect first, then quarantine, then fix.

• How to Fix Flaky Tests?
• How to Avoid Flaky Tests?
• How Test Observability Reduces Flaky Tests?
• How to Handle Inconsistent Behavior When Running Scripts on Different Systems or Environments?