Quarantine Test: How to Isolate Flaky Tests in CI

What Is a Quarantine Test?

A quarantined test is one that has been moved out of the set of checks required to merge, while continuing to execute and report on every run. It still tells you what it finds. It just cannot stop anyone shipping while it does.

The distinction that matters is between a test's result and a test's authority. Quarantine changes only the authority. Everything else about the test, including its assertions and its schedule, stays where it was, which is why quarantine preserves the option to bring it back.

The reason this is worth a formal process rather than an ad-hoc decision is how much CI noise turns out to be non-deterministic. In a study of flaky builds in GitHub Actions across 1,960 open-source Java projects, Ge and Zhang report that 3.2% of builds are rerun, that 67.73% of those rerun builds exhibit flaky behavior, and that this affects 1,055, or 51.28%, of the projects studied. Roughly half the projects examined had the problem quarantine exists to manage.

Quarantine vs Skip vs Delete vs Retry

All four remove the red from your pipeline. They differ in what they leave behind, and only one of them leaves you able to make a decision later.

Retry deserves the sharpest warning of the four, because it is the default in most pipelines and it looks harmless. A retry policy converts every intermittent failure into a pass, including the ones caused by a real race condition in the product. Quarantine at least records that the test is untrustworthy; a retry records nothing at all.

When Should You Quarantine a Test?

Write the entry criteria down before the first argument about a specific test, because in the moment the pressure is always to unblock the release rather than to be rigorous. A workable rule has four parts.

• Repeated, not single - The test has failed more than once, non-deterministically, within a defined window. One failure is a candidate for investigation, never for quarantine.
• Confirmed non-deterministic - It has passed and failed against the same commit. This is the step that separates a flake from a regression, and it is the one most often skipped.
• Not covering a critical path - Payment, authentication, and data-integrity checks warrant a fix or a release hold rather than a quarantine, because the coverage gap is the risk.
• Has an owner and a date - Both assigned at entry. A quarantine request without a name attached is a request to delete the test slowly.

Thresholds circulate as folklore, usually some variant of a failure rate over a fortnight. Pick numbers that match your own run frequency rather than copying them: a suite running 200 times a day and one running twice a week need very different windows to observe the same number of failures. What matters is that the threshold is written down and applied consistently.

How Do You Know It Is Flake and Not a Regression?

This is the question the whole practice turns on. Quarantining a flaky test buys time; quarantining a real defect ships the bug and silences the alarm that would have caught it.

The discriminator is straightforward and frequently got wrong. Rerun against the same commit, not against the latest branch. If the code is identical between attempts and the outcome differs, the test is non-deterministic. If the outcome is stable on that commit and only differs from an older one, something changed and the test is doing its job.

# Wrong: the branch moved between attempts, so a pass proves nothing
git checkout feature-branch && npm test -- --grep "checkout total"

# Right: pin the exact commit that failed, then repeat on identical code
git checkout 9f2c1ab
for i in $(seq 1 15); do
  npm test -- --grep "checkout total" >> rerun.log 2>&1
  echo "run $i exit=$?" >> rerun.log
done
grep -c "exit=0" rerun.log   # mixed results here means non-deterministic

Root-cause categories are worth knowing before you start, because they narrow where to look during the confirmation step.

An empirical study of flaky tests in SAP HANA by Berndt, Bach, and Baltes analyzed 559 fixed-flakiness issue reports and found concurrency the most common category at 23%, or 130 of 559 reports. The same work reports that different test types face different flakiness challenges.

Read a category distribution from someone else's codebase as a starting hypothesis rather than a conclusion about yours. It tells you where to look first, not what you will find.

How Much Does a Healthy Test Vary Between Runs?

Enough to be mistaken for flakiness, which is a good reason to measure your own baseline before quarantining anything on a timeout.

Method. We ran one check twenty times in a row on TestMu AI Browser Cloud: open a page on the Selenium Playground in a fresh Chrome session and wait for a named button to attach. Same target, same assertion, same machine, executed sequentially. The test is not flaky and never failed.

20 runs, same check, same target

  passed        20 / 20        (0% failure rate)
  fastest       12,890 ms
  slowest       19,835 ms
  mean          15,627 ms
  median        15,210 ms
  std dev        2,017 ms
  spread         1.54x  (slowest / fastest)

A test that passed every single time still varied by nearly seven seconds between its fastest and slowest run. None of that variance came from the test or the application. It came from session startup, network, and scheduling.

The consequence for quarantine decisions is direct. Apply a timeout to this distribution and a perfectly healthy test acquires a failure rate out of nothing:

In this TestMu AI Browser Cloud run, a timeout one second above the mean turns a test with a genuine 0% failure rate into one that fails roughly a third of the time. A team watching that test would reasonably call it flaky and quarantine it, and the test would never have been the problem.

Two things follow. Set waits from the observed tail rather than the average, because the average is the one value guaranteed to fail about half the time. And before quarantining on timeout failures, measure the check's own distribution first: if the spread is this wide on a stable test, the flake may be in your threshold rather than in your suite. This is a single check on one account and the absolute numbers will differ on yours, which is exactly why the baseline is worth measuring rather than assuming.

What Does the Quarantine Workflow Look Like?

Five steps, and the last one is the step teams omit, which is why quarantine lanes grow.

• Identify - Surface candidates from failure history rather than from memory. The tests that fail most often across all runs are rarely the ones people complain about most loudly.
• Confirm - Rerun on the pinned commit and establish that the failure is non-deterministic before anything is moved.
• Move - Take the test out of the required checks and into a non-gating lane, recording the owner, the date, and the observed failure rate at entry.
• Track - Keep running it on the normal schedule and keep its results visible. A quarantine lane nobody reads is indistinguishable from a delete.
• Reinstate or remove - Return it to the blocking lane once the root cause is fixed and it has held a run of consecutive passes, or delete it deliberately if the coverage turned out not to be worth the maintenance.

Step one is where tooling helps most. TestMu AI's Test Insights aggregates execution records across builds and configurations and surfaces consistently-failing tests through failure-frequency analysis, which is a more reliable candidate list than anyone's recollection of last week's reds. Its root-cause analysis correlates network, console, and framework logs to localize a likely cause, and that output is a lead to verify during step two rather than a verdict that settles it.

How Does a Test Get Out of Quarantine?

Two conditions, and the first is not optional. A fix has to be identified and shipped. Absence of recent failures is not evidence of a fix, because an infrequent flake passes long stretches by chance, and that arithmetic is worth being explicit about.

Those figures are the binomial probability of observing no failures in a run of that length at each rate. The practical reading is that the required streak depends on how often the test was failing when it entered, which is precisely why the observed failure rate should be recorded at entry rather than reconstructed later.

What Does Quarantine Actually Cost You?

Most write-ups on this topic stop at the workflow, which leaves out the part that decides whether the practice is safe. A quarantined test is a coverage gap with a friendly name.

For the duration, the behavior that test asserted is unguarded. A regression in that area will not fail the build, and it will not fail review either, because the reviewer sees green. That is an acceptable trade for a fortnight on a peripheral feature and an unacceptable one indefinitely on a checkout flow.

Two numbers keep this honest, and neither is the count of quarantined tests. Track the age of the oldest entry, which tells you whether the process drains, and the ratio of entries to exits over a quarter, which tells you whether it is a queue or a landfill. A lane of forty tests that turns over monthly is healthier than a lane of five that has not moved since March.

It is also worth naming what quarantine does not fix. The test is still unreliable, the underlying race or timing assumption is still in the code, and the coverage is still absent. Quarantine buys time to do the work; it is not the work. Our guide to flaky tests covers diagnosing the causes, and managing flaky tests in automation covers the surrounding remediation process.

Governance and the Ways Quarantine Goes Wrong

The failure modes are predictable enough to design against in advance.

• The permanent lane - Tests enter and never leave, because no deadline was set at entry. Enforce it by failing the build when a quarantined test passes its expiry date, which is the one place a gate genuinely belongs.
• Quarantining the messenger - A real intermittent defect gets isolated because nobody reran the pinned commit. The confirmation step is the entire safeguard, and skipping it under release pressure is how it fails.
• Silent growth - The lane expands a test at a time and nobody sees the total. Report the count, the oldest age, and the entry-to-exit ratio wherever the team already looks at CI health.
• Automatic quarantine on a threshold - Convenient and eventually wrong, because a genuine regression that fails intermittently will trip the same rule. Automate the detection and the proposal; keep a person on the decision.
• Quarantine as a substitute for capacity - When the lane grows faster than anyone can repair it, the real constraint is engineering time rather than test quality. Our guide to QA bottlenecks covers identifying which constraint you are actually hitting.

Conclusion

Write the entry and exit criteria before you quarantine the next test, and make both a single page: what qualifies, who owns it, when it expires, and what it has to demonstrate to come back. Teams that skip this step do not avoid quarantine, they just do it informally and permanently.

The one rule worth carrying out of this: confirm non-determinism on a pinned commit before moving anything. Every other mistake here is recoverable, and quarantining a real regression is the one that ships a bug while removing the thing that would have caught it.

To build the candidate list from evidence rather than recollection, the getting started with HyperExecute documentation covers running a suite with per-test reporting and retries configured explicitly, so the failure history you quarantine from is complete.