AI-Powered Testing Solutions for Resolving Flaky Tests

Everybody Hates Flaky Tests

Flaky tests – also often referred to as unreliable or non-deterministic tests – are so commonplace that smart enterprise development teams have put in protocols on how to handle them. (More on this in a bit.) Even so, many developers and testers treat flaky tests as an unavoidable cost of doing business.

Flaky tests were one of the major concerns cited in our Future of Quality Assurance survey of more than 1,600 QA professionals from around the globe. Respondents reported they spend about 8 percent of their time on flaky tests. That is almost as large a time commitment as the 10.4 percent they give to setting up and maintaining test environments.

An ACM-published study of build data from large corporations, including Microsoft and Google, recorded flakiness in their testing as often as 41 percent of the time. These tests were extremely flaky – about half of the jobs that failed ran successfully on a second attempt after being restarted manually. Our survey showed that more than 24% of large organizations get non-deterministic results on more than 5% of their tests.

And that doesn’t take into account the toll that delays and frustration can have on your team dynamics and overall efficiency.

Key Takeaway: QA professionals in the Future of Quality Assurance survey reported spending about 8 percent of their working time on flaky tests, close to the 10.4 percent spent setting up and maintaining test environments.

The Types of Flaky Tests

To develop a plan for tackling flaky tests, you first need to understand the general categories in which they occur.

Random: Just like it sounds. A test fails; you manually restart it; and it completes successfully, with no changes in code or environment. Or it fails and then succeeds on a second run. This is probably the most common kind of test flakiness, and it can drive you crazy as you hunt down possible causes.

Environmental: A test may work on your machine but fails on another system. Or, perhaps more troubling, code fails in a continuous integration (CI) environment.

Branch: The test succeeds within the featured application branch, but then fails when merged into the main. These are slightly less maddening than purely random fails – at least you have a starting point to hunt for inconsistencies and conflicts.

Key Takeaway: Random, environmental and branch flakiness need different fixes, so classifying a flaky test before debugging it prevents hours spent on the wrong investigation.

Flaky Tests Hurt Every Aspect of Software Development

Flaky tests are not a niche annoyance. Removing them has become an industry-wide focus, because flakiness has turned into part of the daily grind for most developers.

Why are flaky tests so awful?

They drain resources: Writing, executing, and re-running tests consume a huge chunk of developers’ time – in some cases, testing consumes more of their time than actually writing code. Flaky reruns are a time-consuming part of that work cycle, and they pull developers away from feature development and ideation.

They delay releases: Resolving unpredictable test results can push back milestones and completely derail a project timeline. That’s good for no one.

They can really stress out developers: Retaining engineering talent is already hard. Going back after a "successful" rollup to hunt a bug that was never real is demoralizing, and it wears down the people who have to do it.

Can undermine the perceived value of the test suite and team: Like everybody else in IT, testing units have to justify the expense of their tools and talent. Compiling a lot of false “fails” in your testing program is a sure way to damage your reputation. Shopify famously lifted the pass rate on their extremely complex development channels to well over 90% by just re-running initially failed tests that had flaked out.

They can lead to buggy releases: If developers and testers don’t trust test results, they may get in the habit of simply pushing out code that has legitimate issues.

Key Takeaway: Flaky tests drain engineering time, delay releases, demoralize developers and erode trust in the test suite, and a distrusted test suite eventually lets real defects reach production.

Why Do They Keep Happening?

So, why do carefully written tests flake out? These are the most common issues that cause non-deterministic tests.

Process timing / timeouts: Testers often write in sleep statements to allow time for the application to complete a request. If the application takes longer than expected, the test will fail. This is particularly tricky if the app is calling to an outside data store or resource that may not always perform as the tester expected. And, if the test environment is getting hammered, that can also slow responses to below the anticipated level.

Concurrency: Tests often expect an EXACT sequence of events, even though the code is written to allow for multiple execution sequences, particularly if different threads are handling the actions. If the test can accept only one sequence, it will fail.

Test order dependency: If testing changes data stores, memory or other aspects of the environment, running subsequent tests out of sequence will ensure failure. Your tests must be able to run independently in any order, as well as clean up after themselves to ensure stability for the next test.

Exotic code/environment conditions: Sometimes coders or environment admins create conditions (intentionally or unintentionally) that are hard to anticipate in test design, but can cause a test to fail under anomalous conditions. Everyone’s heard of the 500-mile email problem at this point – a server upgrade implemented an extremely tight timeout window that was causing emails to fail. This is rare, but it can happen.

Bad tests: Human error is always a real risk. Solid tests should include assumptions that cover all operational bases, as well as measures to enforce those assumptions. Edge cases do happen, and you need to test for them.

Flaky tests can be caused by various factors, including process timing, concurrency, test order dependency, and exotic code or environment conditions, as seen above. these issues can lead to non-deterministic behaviour in tests, making them unreliable and frustrating for developers to deal with.

To understand engineer’s perspectives on flaky tests, watch the complete video tutorial and see how these perspectives can help handle and resolve flaky test issues effectively.

Key Takeaway: Hard-coded sleep statements, concurrency assumptions, test order dependency, exotic environment conditions and weak assertions account for most non-deterministic test results.

How to Tackle Flaky Tests

As we mentioned earlier, flaky tests are such a big problem for software developers and testers that most enterprise teams have developed some protocols for handling them.

Here, Mythili Raju and Harshit Paul have laid out a framework for flaky test mitigation in our LamdaTest Learning Hub, which we strongly recommend you read.

In summary, we suggest that your team:

• Implement consistent test retry mechanisms to ensure that a “fail” is a fail
• Regularly maintain and update your tests
• Encourage open communication between developers and testers
• Set and monitor test performance metrics and KPIs
• Create a weighting scale to prioritize the resolution of flaky tests based on business value
• Constantly gather and analyze test performance data

It’s with the last initiative, analyzing testing data, where the emerging category of AI and machine learning testing tools can have an enormous impact on not only resolving flaky tests but also preventing them from happening in the first place.

For a broader look at how intelligent tools are transforming quality assurance beyond flaky test detection, explore this comprehensive guide on AI in software testing, which covers test generation, execution optimization, and predictive analytics.

Key Takeaway: Consistent retry rules, regular test maintenance, tracked test performance metrics and a business-value weighting for fixes turn flaky test handling into a repeatable process instead of an ad hoc bug hunt.

How Do CI Platforms Quarantine Flaky Tests Now?

Quarantine takes a test with a known flake rate out of the pass or fail decision for a build. The test still runs and still reports, but its result no longer blocks a merge.

Quarantine used to be a script each team wrote for itself. It is now a shipped feature in the test platforms that sit beside CI, and the newer implementations attach an agent to it.

• Auto-quarantine: Trunk tracks a per-test flake rate and quarantines a test that crosses it, without a change to the test code.
• Failure fingerprinting: Trunk groups the different ways a single test fails, so a team reviews one root-cause pattern instead of a column of unrelated red builds.
• CI and framework coverage: Trunk reports flakiness from Jest, Pytest, Playwright, Cypress, Mocha and GoogleTest running on GitHub Actions, CircleCI, Jenkins, GitLab, Buildkite, Semaphore and Harness.
• Bits Code pull requests: Datadog Flaky Test Management groups failures by type, then Bits Code reads execution traces, generates a fix, checks it against existing CI logic and opens a draft pull request.
• High-confidence limit: Datadog restricts Bits Code to high-confidence flakiness issues to hold down noise, and a developer still reviews and merges every generated fix.
• Environment stability: Quarantine absorbs a symptom, so an integrated enterprise test execution environment that drifts between runs keeps producing the failures quarantine is meant to hide.

Quarantine buys back a trustworthy red build, but it is a holding pen and not a fix. A test left quarantined indefinitely is coverage the suite has quietly stopped providing, so treat the quarantine list as a backlog with an owner and a deadline.

Key Takeaway: Quarantine stops a known-unstable test from blocking a merge, but a quarantined test still has to be fixed or the test suite silently loses the coverage that test provided.

AI-Based Tools and Flaky Test Detection

AI tools are great at identifying flaky tests, allowing developers and project managers to prioritize legitimate code errors that can be remediated and advanced toward release while moving flaky test results along a different path.

AI and machine learning can also analyze test results to find the underlying issues, such as environmental factors, that can contribute to ongoing flakiness.

The major strengths of AI are:

Root Cause Analysis: In addition to flagging flaky tests, AI-based tools can parse your testing logs to recognize patterns in non-deterministic results, most often with environmental factors. With these insights, you can resolve timing and resource issues that are derailing your testing efforts. (Of course, AI pattern recognition can also find recurring code errors, but that’s not really “flaky” – it’s just another way that AI tools can improve your software pipeline.)

Adaptive Test Maintenance: AI tools can sort out outdated or unnecessary use cases, which majorly contribute to overall “flakiness” in your testing suite.

Predictive Analytics: AI tools can help development teams avoid flaky environmental factors, based on historical data and fixed deltas you provide the system. No flaky errors, no time wanted.

Continuous Improvement: AI-based tools can support your team as they refine testing strategies for both the near and long term. Software testing should be both comprehensive and efficient, and Big Data analysis of your testing logs can keep you on that path.

With the rise of AI in testing, its crucial to stay competitive by upskilling or polishing your skillsets. The KaneAI Certification proves your hands-on AI testing skills and positions you as a future-ready, high-value QA professional.

Key Takeaway: Root cause analysis, adaptive test maintenance, predictive analytics and continuous improvement are the four ways AI-based tools cut flakiness instead of hiding it behind a retry.

AI Helps Tackle Flaky Tests at the Root

Flaky tests are a serious drain on your development and testing teams’ resources and morale. A solid remediation plan, including AI-powered testing suite tools to diagnose and resolve underlying issues that cause flaky tests, is essential in keeping your projects on schedule and up to the quality standards you demand. The diagnosis part of this process is where AI debugging plays a key role — intelligently analyzing failure patterns to pinpoint why tests behave inconsistently across runs.

To conquer your flaky tests, tools like TestMu AI’s Test Intelligence help your teams to take data-driven actions in identifying, resolving, and preventing flaky tests. By leveraging machine learning and intelligent analysis, TestMu AI aims to enhance the reliability and effectiveness of automated testing, resulting in more robust software delivery.