Webinar: Beyond Convenience: Regression Testing in its Real Avatar [Voices Of Community]

Nuances of Regression Testing

Rahul starts the session with a thought-provoking question to the audience:“Regression testing. When you hear this term, you may instinctively deem it a basic tenet of the software testing realm. But isn’t it curious how we tend to oversimplify things when observed from a distance?”

Rahul further explains that viewing a painting from afar might make you believe it’s a simple task, requiring only a few brush strokes. Yet, when you approach it closely, you begin to discern the intricate details, precision, and effort of every masterpiece.

Upon diving into regression testing, you soon realize that what seemed like a quick task is anything but. There’s an art to it; like any art, it requires attention to detail, methodical precision, and an understanding of the broader picture. So, where are we erring? What crucial aspects of regression testing are we overlooking in the name of convenience or simplicity?

The Reality of Basics

Rahul addressed a concerning industry trend that tends to undermine regression testing. For instance, phrases like “Regression testing doesn’t find bugs,” or the inappropriate application of the ‘Pesticide Paradox’ principle to discredit regression testing have become alarmingly common. This principle, well-articulated in Boris Beizer’s book, is now being taken out of context, leading to skewed perceptions.

Test automation is also a major talking point. It’s an assumption that if tests are automated, they are considered the essence of the regression testing cycle. But, as many seasoned testers know, automated tests can’t capture the intricacies and edge cases inherent in exploratory testing.

Further, the positioning of exploratory testing as a superior alternative and the diminishing value of regression testing over time have added to the confusion. Often, there’s a mistaken belief that regression testing is merely the repetition of known information or written test cases.

The bias toward UI or functionality is a significant issue in the current testing narrative. In many circles, regression testing is seen merely as an extension of functional testing or, at most, some elements of UX testing. This reductionist view needs to consider the multi-faceted nature of regression testing.

In its true essence, regression testing is a layered and nuanced domain. It’s not merely about repeating tests or checking functionality. It’s about understanding the system’s behavior, identifying potential risk areas, and ensuring that new changes have kept existing functionality intact. As we continue to navigate the evolving landscape of software testing, we must revisit our understanding of “basic” concepts and appreciate their depth and importance.

Misconceptions about Regression Testing

Fundamental Confusion

The terms “regression testing” and “regression” are often used interchangeably. This is more than just a gap in foundational understanding; it represents a complete absence of it. Misusing these terms reflects a profound misunderstanding of testing fundamentals.

Oversimplification & Blind Repetition

Regression testing is frequently misconstrued as mere repetition without more profound thought or analysis. This mistaken belief leads to the idea that by simply investing in platforms, tools, and technologies, one can solve all issues related to regression testing. Whether technologically aided or done manually, the essence of regression testing isn’t mere repetition.

Neglect of Innovation

A prevalent misconception is that regression testing solely involves repeating existing tests. The creative aspect, which includes crafting new tests based on evolving requirements, often needs to be noticed or discussed.

Bias Towards Functionality and UI

A reductive view has taken hold where regression testing is often equated only with functional or UI testing. This narrow perspective does a disservice to the comprehensive nature of regression testing.

Misrepresentation

By presenting regression testing as a weak straw man, it’s made to appear inferior, enabling other methods or tools to be marketed as superior alternatives. This tactic oversimplifies regression testing, reducing it to a surface-level activity.

Artificial Distinctions

Another prevalent fallacy is the separation of regression testing from exploratory testing as if they share no common ground. Some even go to lengths to position exploratory testing as a superior alternative, further muddying the waters.

Point of Direct Contact and Ripple Effect

Rahul drew a parallel between an object (like a stone) hitting the water, and as it plunges, there’s an evident point of impact, causing ripples that extend outward. The central impact is apparent, but what about the surrounding ripples? How far do they reach? And in a philosophical sense, how much of the water is genuinely affected by the stone’s descent?

Theoretically, every drop of water feels the stone’s influence. However, real-world constraints make us ask the practical question: which ripples matter the most?

Now, think of these ripples in the context of regression testing. Here, the stone’s impact represents a change or update, and the ripples symbolize the areas of potential influence or disruption. Just as we can’t feasibly track every ripple in a pond, we’re bound by constraints when it comes to regression testing. There’s a cost to consider, time limitations, the availability of skilled personnel, and the right tools.

The Constraints

• Cost and Time: We often need more resources and schedules that restrict the extent of our regression testing.
• Skills and Knowledge: Do our teams genuinely understand the ‘ripples’ of the software world? Do they know which ones to prioritize and how to measure them?
• Tools: Having the correct tools is essential. Even with the best intentions, addressing certain ripples might be possible with the appropriate tools.
• Processes: Even if you have the tools and personnel, the desired results won’t be achieved if the process isn’t aligned.

When any of these elements are missing or inadequate, the approach to regression testing often becomes a fallback to convenience. People begin settling, labeling their makeshift methods “regression testing.” Unfortunately, this diluted understanding and application is widespread in many organizations.

Back to Basics: What is a Regression?

Rahul further explains regression can be understood as a bug that is either transient or non-transient. This means it could be a permanent bug or a temporary one. It might resolve on its own, or it could persist until specifically addressed. While not overly technical or filled with fancy wording, this definition captures the essence of regression in the context of regression testing more effectively than most explanations out there.

The Nature of Regression Bugs

• Unmasked Problems: A problem might already exist, but it remains unnoticed due to how it was previously used or the absence of further changes. Due to specific changes, this problem can become more prominent and get unmasked.
• New Problems at the Point of Change: It’s a common misconception that the regression will only occur elsewhere if a change is made at point A. However, it’s also possible to have regressions at the same point of change.

What is Regression Testing?

Regression testing is essential in software testing, focusing on detecting regressions – bugs caused by changes in the software. It’s crucial regardless of the tools, technology, or organization size. This testing ensures that new modifications don’t negatively impact existing functions by identifying bugs resulting from any change anywhere in the software.

Beyond Functional Regressions

The common bias towards functional testing in regression testing limits its true potential. There is no hard and fast rule that functional changes, such as code modifications, configuration alterations, or new features, will only lead to functional regressions. A functional change could also result in performance, security, or interoperability regressions. This means there’s a many-to-many relationship between the changes made and the potential quality dimensions they can affect.

It’s important to ask yourself whether your regression testing includes performance and security tests. Often, there is an incorrect assumption that existing tests will cover all potential regressions. However, new changes usually require the creation of new tests, as repeating the same tests can be a flawed approach driven more by convenience than thoroughness.

In the agile world, there’s a tendency to prioritize speed, but to incorporate true flexibility, it’s sometimes necessary to slow down. This allows for proper adaptation to changes before accelerating again. Unfortunately, the desire to be fast often compromises the depth and thoroughness of testing.

Practical Regression Testing: Go Beyond Convenience

Regression testing is acknowledged as a highly complex task in software development. It requires much more than mere repetition of existing tests; it demands the creation of new testing approaches and strategies. The challenge lies in balancing the need for exhaustive coverage with the practical limitations of time and resources.

The Complexity of Regression Testing

• Beyond Blind Repetition: Emphasizes the need to create new testing methods rather than just repeat existing ones.
• Accepting the Challenge: Acknowledges the complexity and essential nature of regression testing in software development.

Coverage and Practical Limitations

• The Paradox of Coverage: As test coverage increases, the feasibility of running all tests decreases, leading to a need for selective testing.
• Educated System vs. Chaos: Highlights the importance of making choices based on a well-informed system to avoid disorder.

The Role of Testers and Team Dynamics

• Testers as Moderators: Testers are crucial in steering the regression testing process, acting as facilitators rather than sole executors.
• Collaboration with Cross-Functional Teams: Stresses the importance of involving a team with representatives from various departments for comprehensive testing.
• Risk Relationship Mapping: The necessity of establishing a clear methodology for capturing risk relationships between different software features.

Effective Utilization of Advanced Technologies

• Beyond Surface-Level Solutions: Concerns about using advanced technologies like AI for superficial problems rather than complex testing challenges.
• Potential of AI in Regression Testing: Advocates for the application of AI to build systems that can diagnose interdependencies and construct risk models.
• Reframing Automation Goals: Suggests a shift from using automation for speed and cost reduction to enhancing test coverage and depth.

The session ended with an interactive Q&A session.