Test Your Website on Real Desktop and Mobile Browsers

Run live and automated cross browser tests on Chrome, Safari, Firefox, and Edge across 3,000+ browser-OS combinations. No setup, no device lab.

Testμ

Proudly Presenting: Testing as a Service [Testμ 2023]

In this session of the Testμ’23 Conference, Mesut Durukal shares his thoughts on presenting testing as a service.

Last Updated on:

When talking about software development, there are multiple things like front end libraries, backend services, and mobile apps to consider. But this time around, we’re diving into a different topic: creating the test itself as a service, aka TaaS.

Software testing is becoming challenging day by day. In addition to the technical stuff, we’re dealing with a rapidly changing development process and a lot of work to cover in a short time. While many teams are working hard to deal with these challenges, we’ve noticed a few things:

  • No team is doing things the same way. Each team has its way of approaching things.
  • Some teams are struggling with issues that others have already solved.
  • There’s a lot of wasted effort because of doing the same things over and over.

So, to address this situation, there is a solution: a centralized test framework development idea. The goal is to bring things into balance, help those struggling with problems that others have already solved, and ensure that the code used for testing across different teams is of high quality.

In this session of the Testμ’23 Conference, Mesut Durukal, QA Automation Engineer at Indeed, shares his thoughts on presenting testing as a service.

Youtube thumbnail

If you couldn’t catch all the sessions live, don’t worry! You can access the recordings at your convenience by visiting the TestMu AI YouTube Channel.

Agenda Overview

Mesut lists down the agenda of the session that includes:

Agenda Overview
  • Preparation for Solution Development: Before delving into the solution development phase, it’s important to set the stage and make preparations. This stage occurs even before crafting the actual solution. The goal here is to analyze the problem and lay the groundwork for efficient implementation.
  • Identifying the Problem: In larger organizations, it’s common to face challenges that other teams may have encountered as well. Often, other teams have already tackled similar issues, but due to the sheer size of the organization, sharing solutions isn’t always straightforward. This can lead to various teams independently solving the same problem, resulting in duplication of effort and time. Furthermore, without a standardized approach, code quality can suffer.
  • Addressing the Challenge: To explain this issue, Mesut takes a simple scenario of a test case of locating a button on a webpage, click it, and verify the button label matches an expected string. Surprisingly, even for this uncomplicated case, there were least four different ways to implement the test. These differences can arise regardless of the programming language or technology we’re using. The example code provided uses the Cypress framework, but the broader point remains: there are multiple paths to achieve the same goal.

Problem Statement

Mesut highlights a common problem encountered in large organizations: the challenge of information sharing and the resulting duplication of efforts. He emphasizes that when teams within a sizable organization face problems, they often attempt to solve these issues independently without realizing that other teams may have encountered and solved similar problems before.

Problem Statement

This lack of awareness and communication leads to the duplication of efforts, inefficiencies in problem-solving, and potential code quality issues. The primary problem Mesut highlights is the need for a more streamlined and collaborative approach to addressing common challenges within the organization.

The Proposal- Central Test Automation Framework

As organizations expand and projects multiply, testing practices often encounter challenges like duplicated efforts, inconsistent approaches, and prolonged implementation times. To address these issues, Mesut proposes a solution: the implementation of a central test automation framework.

In the scenario Mesut presents, testing teams face a common challenge. Each team is tasked with creating test cases from scratch for various projects. This results in duplicated efforts, discrepancies in testing methods, and extended implementation timelines. The lack of standardized practices and shared knowledge hampers collaboration and productivity across the organization.

Mesut then explains how a central test automation framework offers a way to overcome these challenges. This framework introduces a standardized approach to testing by providing a repository of pre-built functions and features that can be utilized across different projects and teams. This approach is akin to “testing as a service,” where a package of common functions and tools is shared among teams, facilitating streamlined and efficient testing practices.

Benefits of a Central Test Automation Framework

Mesut highlights the advantages of adopting a central test automation framework approach. By implementing the central framework, teams can achieve several benefits:

  • Standardized Implementation: With a shared set of functions and features, the testing process becomes standardized across teams and projects. This eliminates inconsistencies in testing methodologies and ensures uniformity.
  • Reduced Anti-Patterns: The central framework’s functions are curated by testing experts, minimizing the chances of anti-patterns and suboptimal practices. This helps teams avoid pitfalls during implementation.
  • Elimination of Duplication: Teams no longer need to recreate test cases from scratch for every project. Instead, they can use pre-built functions to carry out everyday testing tasks, reducing redundancy and saving time.
  • Enhanced Monitoring: The central framework can include additional features like monitoring capabilities. Teams can track metrics and gather insights during test execution, aiding in performance analysis and continuous improvement.

Key Targets of Implementing Central Testing Framework

Mesut then discusses several targets or objectives related to the implementation of a central testing framework and the benefits it can bring. Here are the key targets he mentions:

Key Targets of Implementing Central Testing Framework
  • Standardization: One of the primary objectives is to establish a consistent method for implementing and executing test cases. With a centralized framework, teams can adhere to the same methods and practices, ensuring uniformity across projects.
  • Code Quality: Next target is to enhance code quality by implementing comprehensive code analysis and quality checks within the central framework. This objective involves identifying and rectifying code quality issues and anti-patterns.
  • Duplication Reduction: Another target is to eliminate redundant efforts by offering a common solution. Teams will no longer need to create their own implementations for common problems, reducing duplication of work.
  • Monitoring and Metrics: Mesut suggests that the central framework can include monitoring features. Teams can monitor various metrics during test execution, enhancing visibility into testing activities and aiding in the identification of areas for improvement.
  • Traceability: Through the use of the centralized framework, teams can improve traceability in testing activities. This involves enhancing the ability to trace and comprehend the implementation and execution of test cases.
  • Collaboration: The central framework aims to foster collaboration among teams by providing a shared solution. It encourages the exchange of knowledge and best practices across the organization.

Roadmap to Achieve Targets

To achieve the vision of a central test automation framework, Mesut emphasizes the importance of a well-defined roadmap. The roadmap encompasses key stages:

Roadmap to Achieve Targets
  • Defining Objectives and Key Metrics: Begin by setting clear objectives and metrics to measure success. For instance, measuring the time spent on implementing test cases can gauge the framework’s impact on efficiency.
  • Gathering Requirements: Collect requirements by identifying common challenges faced during testing. These requirements will guide the features and functions to be included in the central framework.
  • Prioritization: Prioritize requirements based on their impact and urgency. Decide which features to implement first and establish a sequential order for implementation.
  • Building the Framework: Develop the central framework by implementing the chosen features and functions. Ensure that the framework aligns with the defined objectives and metrics.
  • Quality Assurance: Validate the framework’s functionality and effectiveness through testing and quality assurance measures. Ensure that it meets the desired standards and delivers the intended benefits.

Furthermore, Mesut underscores the significance of continuously tracking metrics and assessing the framework’s impact.

According to him – by comparing expected results with actual outcomes, teams can determine the framework’s effectiveness and identify areas for improvement. Through transparent and continuous monitoring, teams can ensure that the central framework is delivering on its promises of efficiency, standardization, and streamlined testing practices.

Objectives and Key Metrics

Talking about objectives and their metrics, Mesut shares the significance of defining clear objectives and key metrics when implementing a centralized testing framework. Here are the key points he makes about objectives and key metrics:

Objectives and Key Metrics
  • Measurable Goals: Mesut stresses the importance of setting specific, measurable, achievable, relevant, and time-bound (SMART) objectives. These objectives should be clear and quantifiable, allowing teams to assess whether they are achieved.
  • Motivation: He highlights that objectives serve as motivation for the implementation of the framework. Teams should have a clear understanding of what they are trying to accomplish by using the central testing framework.
  • Quantitative Measurement: Mesut suggests identifying key metrics that can quantitatively measure the success of the framework. These metrics help in evaluating whether the objectives are met.
  • Comparison: He mentions the need to compare the expected situation with the actual situation. Metrics should provide a basis for comparison, allowing teams to gauge the impact of the framework.

Mesut provides examples of key metrics related to quality and efficiency. For quality, he mentions metrics like test coverage, the capability of test cases to find issues and false alarm ratios. For efficiency, he discusses metrics such as automation ratio, execution duration, and implementation duration.

  • Continuous Tracking: He underscores the importance of continuously tracking these metrics. Metrics should not be an one-time assessment but should be monitored over time to assess the ongoing impact of the framework.
  • Transparent Measurement: Mesut recommends that metrics should be transparent and visible to all stakeholders. This transparency ensures that everyone can see the progress and impact of the framework.
  • Objective Evaluation: He suggests that metrics should evaluate different dimensions of quality, such as code quality and execution speed. This helps in understanding the framework’s effect on various aspects of testing.

Challenges While Implementing Centralized Test Framework

Mesut takes the session ahead and discusses the challenges faced when implementing a centralized testing framework and how these challenges can be addressed. Here are the key points he makes about challenges:

Challenges While Implementing Centralized Test Framework
  • Understanding Common Problems: Mesut highlights the challenge of identifying common problems faced by different teams or individuals within the organization. These problems may include issues related to testing, test automation, code quality, and more.
  • Variability in Solutions: He mentions that teams often come up with their own solutions to similar problems, leading to variability in approaches and implementations. This variability can result in duplication of effort and code quality issues.

Mapping Challenges to Requirements:

  • Requirement Analysis: Mesut suggests that the identified challenges should be mapped to specific requirements for the centralized framework. Each challenge should have a corresponding requirement that the framework aims to address.
  • Prioritization: Once challenges are mapped to requirements, prioritization becomes essential. Teams should prioritize these requirements based on their impact and the degree to which they address common challenges.

Solving Challenges:

  • Implementation: To overcome challenges, Mesut recommends implementing features and components within the centralized framework that directly address the identified requirements. These features should provide solutions to the common problems.
  • Continuous Improvement: He emphasizes that the framework should not be static but should evolve and improve over time. As new challenges arise or requirements change, the framework should adapt to meet these evolving needs.

Feedback Loop:

  • User Feedback: Mesut advocates for maintaining a feedback loop with the teams and individuals using the centralized framework. User feedback is crucial for understanding whether the framework effectively addresses challenges and requirements.
  • Iterative Development: He suggests that the development of the framework should be iterative. Teams should be open to making adjustments and enhancements based on user feedback and changing requirements.

Implementation of a Centralized Testing Framework

Mesut covers various aspects related to building a centralized testing framework. Here are the key points he covers:

Flexible Workflow:

  • Reusability: Mesut emphasizes the importance of reusability when developing the framework. The framework should support the execution of test cases across different projects and products, ensuring that the same features can be used even when the system under test changes.
  • Workflow Triggering: He discusses the triggering of workflows within Continuous Integration/Continuous Deployment (CI/CD) pipelines. These workflows are initiated based on triggers such as schedules or code repository changes.

Test Suite Composition:

  • Test Case Selection: Mesut explains how the framework should allow for the selection of relevant test cases to be included in a test suite. Different projects may require different subsets of test cases.
  • Parallel Execution: He highlights the need for parallel test execution, enabling multiple test cases to run simultaneously on different pipelines or virtual machines. This parallelism improves efficiency and reduces execution time.

Test Fixture Usage: Mesut discusses the use of test fixtures, such as “before all” and “before each” hooks, to manage common test steps and eliminate duplication. These fixtures help in setting up and tearing down the test environment.

Artifact Collection and Reporting:

  • Visibility: He emphasizes the importance of collecting artifacts and evidence during test execution. This includes screenshots, logs, and other relevant data that can help in diagnosing test failures.
  • Reporting: Mesut mentions the need for a centralized dashboard or reporting mechanism where test execution details, including success and failure information, can be viewed and analyzed.

Technology and Framework Selection: He highlights the significance of selecting the right tools and technologies for test automation. This includes choosing the appropriate programming language, frameworks, and libraries based on project requirements.

Quality Dimensions: Mesut discusses various non-functional quality attributes that should be considered during implementation, such as flexibility, integrability, scalability, reliability, performance, understandability, and readability of the code.

Benchmarking: He recommends conducting benchmarking studies to evaluate different tools and frameworks. These studies help in selecting the tools that best meet project requirements.

Objective Evaluation: Mesut underscores the importance of objective evaluation metrics to measure the impact of the framework. Metrics should provide quantitative data on aspects like code quality and execution speed.

Testing as a Service

In the last phase, Mesut discusses the benefits and advantages of the centralized testing framework and how it serves the needs of various teams and projects within the organization as a service.

Testing as a Service

Motivation and Goals: Mesut reiterates that the primary motivation behind the centralized framework is to expedite the implementation of test cases and testing activities.

Tracking and Analysis:

  • Analyzing Implementation Components: He explains how he analyzed the components of the test case implementation by breaking them down into tasks, such as defining locators, implementing test steps, and handling user interactions.
  • Ticket Tracking: Mesut mentions using a project management tool like Jira to track these tasks and subtasks related to test case implementation.

Comparative Analysis:

  • Tool Selection Impact: He discusses how he performed a comparative analysis by implementing the same test case using multiple automation tools and frameworks. This analysis helped in understanding the impact of tool selection on implementation time and efficiency.
  • Execution Speed: Mesut also mentions evaluating execution speed when running test cases both locally and in CI/CD pipelines. This evaluation helps in determining the framework’s performance.

Quality Metrics: He highlights the importance of quality metrics, both functional and non-functional, in assessing the effectiveness of the framework. These metrics include code quality, execution speed, and reliability.

Feedback and Iteration:

  • Feedback Loop: Mesut stresses the need for a feedback loop with teams and individuals who use the centralized framework. User feedback is essential for continuous improvement.
  • Iterative Development: He emphasizes that the development of the framework should be iterative, with room for adjustments and enhancements based on feedback and changing requirements.

User-Friendly Framework: Mesut mentions that for the framework to be successful, it should be user-friendly and transparent. Teams should find it easy to understand and use the framework effectively.

Closing Note

Mesut concludes the session by summarizing the main points discussed, emphasizing the importance of a centralized test automation framework for addressing common problems across teams in a large organization, eliminating duplication of effort, and improving efficiency.

Finally, he encourages attendees to ask their questions on the LambdaTest Community or reach out to him via LinkedIn for further discussion.

Tracking tests across tools and teams? TestMu AI's Test Manager keeps test cases, runs and results in one place.

To run your first test, start free, or
for a full walkthrough of the product.

Author

...

TestMu AI

Blogs: 281

  • Twitter
  • Linkedin

TestMu AI is World's First Full Stack AI Agentic Quality Engineering platform that empowers teams to test intelligently, smarter, and ship faster. Built for scale, it offers a full-stack testing cloud with 50K+ real devices and 3,000+ browsers. With AI-native test management, MCP servers, and agent-based automation, TestMu AI supports Selenium, Appium, Playwright, and all major frameworks. AI Agents like HyperExecute and KaneAI bring the power of AI and cloud into your software testing workflow, enabling seamless automation testing with 120+ integrations. TestMu AI Agents accelerate your testing throughout the entire SDLC, from test planning and authoring to automation, infrastructure, execution, RCA, and reporting.

Add to Google preferred sources

Summarise with AI

Copied to Clipboard!
...

3000+ Browsers. One Platform.

See exactly how your site performs everywhere.

Try it free
...

Write Tests in Plain English with KaneAI

Create, debug, and evolve tests using natural language.

Try for free

Did you find this page helpful?

More Related Blogs

TestMu AI forEnterprise

Get access to solutions built on Enterprise
grade security, privacy, & compliance

  • Advanced access controls
  • Advanced data retention rules
  • Advanced Local Testing
  • Premium Support options
  • Early access to beta features
  • Private Slack Channel
  • Unlimited Manual Accessibility DevTools Tests