Master Your Test Suite: Unveiling Enhanced Flaky Test Detection & Customization!

So How Does Flaky Test Detection Work?

Flaky Test Detection employs WebDriver command logs to pinpoint flaky tests identified with a flake icon. These command logs represent the steps executed by the test script during the test.

The detection process involves an algorithm that identifies tests showing inconsistent behavior, marked by changes in their status within a sequence of test runs. The threshold of the flakiness depends on the transition between statuses (from Passed to Failed or vice versa) based on user-specified configurations.

For instance, a test sequence showing Passed -> Failed -> Passed indicates two status transitions in the three most recent executions. The algorithm also analyzes individual test commands and their statuses, providing a detailed view of potential flakiness sources at the command level.

To detect flakiness, it’s important to identify where it occurs. Watch this video to gain insights into the common sources of flakiness.

Note: Currently, this feature only supports Selenium-based tests for Web Automation and HyperExecute subscribers.

Customization of Flaky Test Detection

With the revolutionary Custom Flaky Test Detection, you can customize and manage flaky tests effortlessly. With options to adjust grouping mechanisms, sliding window size, and flake rate thresholds, our updated settings offer a tailored detection process to meet your unique testing needs, ensuring a streamlined and efficient testing experience.

Now let’s learn how you can use the Custom Flaky Test Detection settings:

• Grouping Mechanism: The grouping mechanism in Flaky Test Detection allows you to set the criteria for how tests are categorized for flakiness. By default, tests are grouped by the testing environment details provided in the capabilities, such as test name, browser, operating system (OS), and resolution.

While the test name is the main grouping criterion, you can adjust the grouping to meet your needs by selecting or deselecting environment variables. This customization lets you define test categorization with precision.

• Sliding Window: The sliding window feature considers a set number of previous tests when evaluating a test’s likelihood of being flaky. By default, it analyzes the last 10 tests. Adjusting this number allows you to customize the amount of past data that should be used in the evaluation. A larger sliding window offers a broader view of historical data, helping to identify subtle trends or variations in test results.
• Flake Rate Threshold: The flake rate threshold determines when a test is labeled as flaky. Adjusting this rate allows for precise control over flakiness detection sensitivity. A lower threshold will identify more flaky tests, capturing even minor inconsistencies, while a higher one offers tolerance to minor variations, highlighting only significant discrepancies in test results.

Carefully choosing the right threshold ensures that the detection of flaky tests aligns perfectly with your specific testing requirements.

Note: Identify and mitigate future test issues with AI-native test intelligence insights. Try Now!

Expanding The Power Of Flaky Test Detection

We are actively working on extending the Flaky Test Detection capabilities to popular testing frameworks like Cypress, Playwright, Puppeteer, and Appium. This expansion will empower users to assess and manage test reliability across a broader spectrum of testing scenarios in web and app-based tests, enhancing the overall testing experience. Stay tuned for updates on the availability of Flaky Test Detection support for these frameworks.

All In All!

In conclusion, our enhanced Flaky Test Detection and Customization tool is a game-changer for managing inconsistent tests in your suite. It not only identifies and provides insights into flaky tests but also allows for detailed historical analysis and customization to fit your specific testing needs. With upcoming support for additional testing frameworks, this tool is set to offer comprehensive assistance for both web and app testing scenarios, making your testing process more efficient and reliable.

Happy Testing!