Transitive Testing [Testμ 2023]

Insights into Software Testing and Reliability

Titus starts the session, and at first, he explains why testing is essential – it’s not just about passing tests; it’s about making users happy. He talks about how the higher-ups, like the board of directors and the CEO, don’t care about testing metrics; they care about users having a good experience on their website. And speed is crucial for that!

He explains the difference between human testers and automated testing. Humans can give us more useful information by looking at different things and providing valuable insights. On the other hand, if we focus too much on automated test coverage, we might miss out on reliability, which is super important.

Titus shares a simple way to look at the costs of test automation: maintenance, creation, and execution costs. He mentions that sometimes people want to speed up test execution without realizing that their tests fail too much. It’s like getting results faster, but those results are useless because the tests are unreliable.

He encourages everyone to focus on reliability and accuracy in testing. He talks about three main principles: doing one thing at a time (atomic testing), ensuring tests can run independently (autonomous testing), and running tests in parallel to speed things up (parallel test execution).

Test Address Book Site: An Example of User Journey

Titus goes ahead and shows his Test Address Book site. The site’s functionality includes features related to authentication and basic CRUD (Create, Read, Update, Delete) operations. Titus points out that this site is a practical example for evaluating how testing code performs because it covers common testing scenarios involving user authentication and data manipulation.

He talks about the concept of a user journey and how it relates to the Test Address Book site. Essentially, a user journey is a way to describe how a user interacts with a website or application. It’s like walking through all the steps a user takes, from the very beginning to completing specific tasks on the website.

In the context of his Test Address Book site, Titus explains that a user journey involves a series of actions or steps when using the site. These actions include signing up, signing in, navigating through the site’s different pages, creating new address entries, editing existing ones, and potentially even deleting entries.

Titus demonstrates the user journey with the help of an example of a real user visiting a website to manage your address book. The journey would start with the user creating an account by signing up and providing details like name, email, and password. Upon signing up, the user will log in, and that’s another step in the journey. Now, he will start navigating through the site’s pages, like viewing address book or adding a new address entry.

If the user wants to add a new address for a friend, he will click on a button or link to create a new address entry, and a form will appear where he can fill in details like friend’s name, address, and phone number. After that, he will save the entry. That’s the user journey for creating a new address.

But it doesn’t end there. In the journey, the user might also need to edit an address. To update the address, the user will click on the addresses he wants to change, make edits in the form, and save the changes.

In some cases, the user might even decide to remove an address if his friend moves out. That’s another step in the journey, where user will click a delete button or link and confirm the removal.

Autonomous – Test Data Management

With transitive testing well understood, Titus takes the discussion toward another critical aspect of modern software development – test data management. He explains four basic approaches to test data management.

The first approach he mentions is “grab and hope.” In this method, a tester simply takes any available data on the site, such as grabbing the first product for testing a checkout process. However, Titus shares how this approach can lead to issues. For instance, repeatedly grabbing the same product might lead to it going out of stock, causing unexpected failures in tests. It’s a risky approach as it doesn’t ensure data consistency.

The second approach is using “static references.” This involves maintaining extensive spreadsheets or files with predefined data for each test. When creating a new test, testers need to populate it with data from this repository. However, Titus points out that this method is cumbersome and doesn’t scale well. It can become challenging to manage large datasets and keep them synchronized.

The third approach is “dynamic fixtures,” which is similar to static references, but data is stored in a database. Tests refer to specific items in the database when needed. Although it’s a step up from static references, Titus noted that it shares similar challenges regarding data management and synchronization.

The fourth and ideal approach, according to Titus, is for each test to manage its data independently. While it may not always be feasible due to constraints like external systems or content management systems, it’s the most maintainable option. In this approach, tests generate their required data dynamically as needed. Titus also mentions using libraries like Faker to generate random data for fields that don’t require specific values.

Titus explains that autonomous tests in the context of test automation are designed to be independent and self-contained. Each test can run individually, in any order, and alongside other tests without affecting their results or execution. This ensures that the results of one test do not impact the results of another. This autonomy allows for efficient and reliable testing.

In addition to autonomy, Titus discusses the concept of “just in time.” He emphasizes the importance of generating test data dynamically and on-demand, rather than relying on static or pre-defined data.

Using tools like the Faker library, test data can be generated in real-time for specific test scenarios. This approach minimizes the need for maintaining large datasets and allows tests to focus on relevant, current data, making them more efficient and maintainable.

Atomic Autonomous Authentication

Titus highlights the importance of an atomic autonomous approach to authentication in test automation. He stresses the importance of creating self-contained and independent authentication tests that can operate without relying on other tests.

Here’s why this matters: By ensuring that authentication tests stand on their own, they become more reliable and maintainable. Users will not encounter situations where a failure in one test disrupts the entire testing process or leads to cascading failures. Each authentication test can be executed independently, and the results remain consistent.

Moreover, Titus recommends using APIs (Application Programming Interfaces) for authentication whenever possible. This means bypassing the user interface (UI) and directly interacting with the backend through API calls to create and authenticate users.

This approach not only saves time but also provides a more robust and efficient way of handling authentication, especially when you need to log in frequently for various tests.

Titus’s Comprehensive Testing Approach

After talking about atomic autonomous authentication, Titus explains a significant transition – shifting from traditional web app testing to API testing.

He explains that modern applications often employ API gateways, serving as intermediaries between user interfaces and databases. By using HTTP clients for testing, he makes the process more efficient and dependable.

One of the key takeaways from Titus’s approach is the benefit of API-based authentication. He highlights how bypassing UI interactions and directly authenticating users through APIs can significantly save time and enhance the reliability of testing processes.

Titus also emphasizes the importance of validating API actions. He uses the concept of transitive properties to illustrate how UI actions can be seamlessly substituted with API actions, ensuring robust and maintainable testing practices.

As the transition to API-based testing occurs, Titus advocates for the inclusion of an additional test. This test serves as a bridge between UI and API interactions, ensuring consistency and reliability throughout the testing process.

Extending this strategy to address-related tests, Titus demonstrates how API interactions can efficiently replace UI actions. This approach enhances overall reliability and efficiency in testing, as it simplifies the testing process and reduces dependency on UI interactions.

Q&A Session!

• Can you discuss the situation where your transitive testing approach enabled you to adapt swiftly to UI changes and maintain the stability of your test suite?

  Titus: The essence of transitive testing is to make your tests resilient to UI changes. When UI changes occur, only one of your tests should ideally fail while the rest continue to pass. This pinpointed failure lets you quickly identify which part of your application was affected by the UI change.

For instance, if there’s a problem with the login form, all tests requiring login might still pass, except for the one that fails, signaling the issue. This enables you to rapidly understand the problem, reproduce it, and take corrective action. In essence, transitive testing minimizes the impact of UI changes on your entire test suite, making it more adaptable and maintainable.

• Can you share your journey from UI-centric testing to more atomic and autonomous testing?

  Titus: My transition from UI-centric testing to a more atomic and autonomous approach evolved gradually. It was shaped through discussions with colleagues and my realization that the more time we spent in the UI, the more we encountered its inherent complexities. Browsers can be unpredictable, and relying solely on UI tests becomes inefficient and unreliable.

The key shift was to leverage APIs to perform as much testing as possible. By doing so, we could focus on testing specific features and their interactions rather than being bogged down by UI intricacies. This transition allowed us to scale our testing efforts and execute tests more reliably in parallel.

• What are some common misconceptions about transitive testing, and how do you address them?

  Titus: Transitive testing is a term I’ve coined, so there might not be widespread misconceptions yet. However, one common misunderstanding in the testing community is that UI tests are the most comprehensive and reliable way to validate an application. Some might believe that testing through the UI provides the most realistic representation of user interactions.

To address this, it’s important to emphasize that the goal of transitive testing is not to replace UI testing but to complement it strategically. Transitive testing allows for faster and more reliable testing of core functionalities and state changes, while UI testing remains crucial for evaluating the overall user experience.

• How does transitive testing contribute to ensuring software reliability and quality?

  Titus: Transitive testing contributes to software reliability and quality by streamlining the testing process and ensuring that tests are more resilient to changes. By isolating specific functionalities and testing them independently, you reduce the chances of cascading failures due to UI changes.

Moreover, transitive testing allows you to focus on the core contract between the API and the UI. When both the API and UI validations align, you can have greater confidence that your software functions correctly. This approach improves reliability because you’re validating essential functionalities more efficiently, and it enhances overall quality by catching issues early in the development process.

• How do you identify and manage transitive dependencies in your testing process?

  Titus: Identifying and managing transitive dependencies in testing can be context-dependent. It often involves evaluating which forms or functionalities in your application rely heavily on setting state and interacting with data.

One approach is creating objects representing the input data for various tests. These data objects can be used in the UI and API testing layers. By ensuring that the data objects match the elements on the page and the fields in the API, you can streamline the testing process. This way, you manage dependencies by focusing on the shared data objects, making your tests more maintainable and less prone to failure.

• How do we limit transitive closure for static regression test selection approaches?

  Titus: Limiting transitive closure in static regression test selection approaches involves determining which tests are essential for maintaining confidence in your software’s quality while optimizing resource usage. It’s about striking a balance between test coverage and resource efficiency.

In practical terms, consider evaluating your test suite regularly and identifying tests that may no longer provide substantial value or are too resource-intensive to maintain. Tests that consistently fail or aren’t aligned with critical functionalities may be candidates for removal or optimization. The goal is to ensure your test suite focuses on the most critical scenarios and provides meaningful information while minimizing resource burden.

• Can you provide examples or case studies where transitive testing has helped uncover critical issues or improve overall test coverage significantly?

  Titus: While I don’t have specific case studies, I can attest to the positive impact of transitive testing based on my experience at multiple companies. Implementing transitive testing has consistently led to improved reliability, enhanced test coverage, and faster test execution.

By isolating core functionalities and validating them through the UI and API, teams have caught critical issues early in the development process. This approach ensures that critical contracts between the UI and API are maintained, leading to fewer regressions and more confidence in the software’s quality.

Have you got more questions? Please put it on the TestMu AI Community.

Testing web or mobile apps without writing code? KaneAI by TestMu AI turns plain-English instructions into tests.

To run your first test, start free, or

book a demo

for a full walkthrough of the product.