Assure your AI deployment - across enterprise and real world

We assure that your AI works as intended. Safely and securely.

Testμ

A Tester’s Journey In The World Of Machine Learning [Testμ 2023]

In this session of the Testμ 2023 Conference, Shivani Gaba takes you on a journey through her experience of testing an ML-based application.

Last Updated on:

Machine Learning (ML) has become really crucial in today’s software world. People are working hard to create systems that use ML. But there’s a question: How can we make sure these applications are good? What can testers do to help?

Usually, testing isn’t given a lot of attention, and testers might not know what they can do. Some people think, “You can’t test machine learning.” But is that true? If not, how can we show that it’s not true? How can testers show they’re useful in the world of ML?

In this session of the Testμ 2023 Conference, Shivani Gaba, Engineering Manager at Beyonnex, delves into finding answers to these questions and more.

She takes us on a journey through her own experience of testing an ML-based application, even though she didn’t have any previous experience with it. She talks about what motivated her, the challenges she encountered, the lessons she learned, and the methods she used to test these kinds of systems.

Youtube thumbnail

If you couldn’t catch all the sessions live, don’t worry! You can access the recordings at your convenience by visiting the TestMu AI YouTube Channel.

Let’s see the major highlights of this session!

Introduction and Motivation

Shivani explains her journey that began four years ago when she worked on a news product. This product allows users to subscribe to different sources, read news, and articles. Her primary role was in functional testing.

One day, during a coffee chat with her boss, they were discussing an incident involving toilet seat recommendations on Amazon. This conversation led them to realize the presence of unexpected bugs and failures in various systems, such as Google’s object recognition misclassifications, Microsoft’s offensive chatbot, and Amazon’s biased recruitment tool. This prompted Shivani and her boss to question who ensures the quality of such products and whether testers play a role in their development.

seat recommendations on Amazon

Challenges Faced and Initial Exploration

Shivani explains that her curiosity led her to ponder over the role of testers in these scenarios. She observed that even their news product, which aims to provide personalized article recommendations, relies on a model to generate suggestions. Despite no dedicated testers in this project, Shivani’s boss challenged her to take on the role due to her testing expertise. While excited, Shivani felt nervous as models were unfamiliar territory to her.

Upon joining her team, Shivani faced the challenge of testing machine learning (ML) models. Accustomed to testing conventional systems with clear expected results, she struggled to define expected outcomes for the ML-generated recommendations.

She uses a poll analogy to highlight the difficulty – while it’s evident that two plus two equals four, predicting whether a user would like a specific article based on their preferences proved much more complex.

Challenges Faced and Initial Exploration

Shivani sought guidance from colleagues and the wider community, including developers. However, she found that many shared her confusion regarding testing ML models. Their uncertainty about the role of testers in this context added to her own questions. Despite this, Shivani remained determined to find a way to test ML systems effectively.

She then turned to Google to understand how to test machine learning models. Shivani discovered terms like “recall, ” “precision, ” and “confusion matrix, ” but found them confusing. Attempting to apply this knowledge, she approached her developer with newfound concepts, only to realize her lack of understanding. This encounter further heightened skepticism from developers about the tester’s role in the project, leaving Shivani disheartened and questioning her approach.

tester's role in the project

Embracing a Quality Assurance Perspective

Shivani continues her discussion and explains how she realized she needed to approach the task differently—She decided to focus on her domain expertise and apply it from a quality assurance perspective.

By combining her product knowledge with a fresh perspective as an end user, she aimed to provide a unique angle that developers might overlook. This approach intrigued her developers and renewed their excitement for her involvement.

She leveraged her skills as a questioner, critical thinker, and product expert to enrich the testing process. By challenging assumptions, introducing novel test ideas, and enhancing testability, Shivani’s efforts resulted in preventing bugs from reaching production and improving the overall quality of the ML-based system.

Embracing a Quality Assurance Perspective

Understanding Machine Learning Basics

Shivani shares that she started by understanding the basics of machine learning. She defines machine learning as the process of enabling machines to learn from data and algorithms without explicit programming. She learned about different types of machine learning, including supervised and unsupervised learning, and discovered that her project used supervised learning for personalized article recommendations.

Understanding Machine Learning Basics

She then delves into how machine learning models learn. She drew parallels to how children learn and explained the three stages of model learning: data collection, feature selection, and model training. She emphasizes the importance of quality data and introduced the principle of “garbage in, garbage out.” Shivani recognizes the role testers could play in ensuring data quality, highlighting areas such as spotting inconsistencies, addressing biases, and handling outliers.

Shivani discusses the significance of data collection and quality assurance. She explains that the quantity and quality of data are crucial for effective model training. She shares practical examples of bugs she encountered in the collected data, such as duplicates, missing values, invalid data, and biases. Shivani stresses upon the tester’s role in collaborating with developers to identify and rectify data-related issues.

Principle of SISO

Shivani explains the principle of the SISO as the guiding concept for her talk—she emphasizes the importance of data quality over just quantity in training models. Testers play a vital role in ensuring data quality by checking for inconsistencies, duplicates, and biases in the data. They can also use tools like Pandas, Matplotlib, and great expectations for visualization and automated checks. Shivani uses an example of an article recommender system to illustrate how testers can contribute to different steps in the process.

Principle of SISO

In the first step, data collection and cleaning, testers help identify and fix bugs, missing values, outliers, and apply domain knowledge. They also addresses privacy, security, and biases. In the second step, feature selection, testers work with developers to review, enhance, and document features. Shivani presents a case where she helped ensure consistency and introduced valuable features.

In the third step, model training, testers verify the training-testing dataset split, check the representativeness of the testing set, and contribute to model evaluation using techniques like precision, recall, and confusion matrices. She discusses how she suggested and fixed an issue related to inconsistent rules.

Shivani then explains the delivery process, where models make predictions for user requests. Testers can validate APIs, test component integrations, and even propose methods for comparing old and new models’ performance.

Example of Article Recommender System

Shivani presents a practical example to illustrate her points. She introduces the data snapshot representing an article recommender system. The data includes user IDs, genders, article IDs, ages, and whether users clicked on articles. She mentions that this example data serves as a basis to demonstrate how testers can identify bugs and issues.

Example of Article Recommender System

She highlights the specific problems found in the example data, such as duplicates, missing values, invalid data (negative age), and outliers. She links these issues to real-world scenarios and the impact they could have on the model’s performance.

Feature Engineering

Shivani proceeds to explain the process of selecting model features. Features are attributes or variables fed to the model for learning. Testers collaborate with developers to review, enhance, and document features. She shares an example where she identified inconsistencies in calculating total article views and clicks, leading to a refinement of the features.

Feature Engineering

Shivani then delves into the training phase of the model. Data is split into training and testing sets, and testers ensure the split ratio is suitable and representative. Evaluation techniques like recall, precision, and confusion matrices help assess the model’s performance. Feature importance analysis aids in identifying significant features. Testers also contribute to testing APIs, model integration, and the rollout of new models.

She then concludes by summarizing how testers contribute to such projects. She emphasizes learning the system, applying testing skills, collaborating with developers, identifying bugs, ensuring data quality, selecting model features, evaluating performance, and participating in the model’s delivery and improvement phases. Shivani underscores the importance of using existing testing skills and learning new ones to effectively contribute to these projects.

Q&A Session!

  • Does machine learning testing require a prerequisite of data science?
    Shivani: As mentioned earlier, in my view, comprehending and testing any system requires a grasp of its fundamental components and the arrangement of building blocks. However, delving into an extensive study of data science isn’t necessary. You don’t have to familiarize yourself with every computation or library or write all the code for it.
  • Rather, it’s akin to any other project: developers create things, and you should possess a foundational comprehension of the project’s nature. This allows you to gain a more profound insight into the project’s structure. While having some knowledge of data science is beneficial, it’s not imperative to possess all-encompassing expertise or an in-depth understanding of it.

  • What if someone doesn’t get the opportunity to work in ML-based model testing? How can s/he learn? In the outside market, they are not preferring to hire someone to test this who doesn’t have experience.
    Shivani: I believe that we have the opportunity to learn extensively from the wealth of open resources available nowadays. Over the past few years, numerous valuable materials have surfaced that can help in your learning and self-training. Additionally, there’s a wealth of informative talks to explore. Notably, you might find exceptional presentations by individuals.
  • In your experience, how can testers make a significant impact on the success of machine learning projects beyond traditional QA practices?
    Shivani: In the realm of innovation beyond conventional methods, I believe it’s crucial to embrace unconventional thinking. In contrast to typical scenarios where you often consider the end user’s perspective and their expectations, here, you might find yourself uncertain about those expectations.
  • Hence, it’s essential to explore novel techniques that might not immediately come to mind. This process requires extensive brainstorming, as some approaches may prove successful while others might not yield the desired outcomes. As illustrated during the presentation, one approach involves relying on collective responses rather than individual ones.

    This methodology evolved through an iterative process, where several strategies didn’t initially yield positive results. Thus, I strongly advocate for active brainstorming and collaborating closely with developers—akin to other projects.

    Such collaboration aids in comprehending the project’s intricacies and identifying areas where unique value can be introduced, diverging from traditional approaches. Additionally, a pivotal aspect is focusing on data: Clean and high-quality data serves as the foundational step in this context.

If you have more questions, feel free to post at the TestMu AI Community.

Testing an AI agent? TestMu AI's Agent Testing validates chat, voice and IVR agents before your customers talk to them.

To run your first test, start free, or
for a full walkthrough of the product.

Author

...

TestMu AI

Blogs: 281

  • Twitter
  • Linkedin

TestMu AI is World's First Full Stack AI Agentic Quality Engineering platform that empowers teams to test intelligently, smarter, and ship faster. Built for scale, it offers a full-stack testing cloud with 50K+ real devices and 3,000+ browsers. With AI-native test management, MCP servers, and agent-based automation, TestMu AI supports Selenium, Appium, Playwright, and all major frameworks. AI Agents like HyperExecute and KaneAI bring the power of AI and cloud into your software testing workflow, enabling seamless automation testing with 120+ integrations. TestMu AI Agents accelerate your testing throughout the entire SDLC, from test planning and authoring to automation, infrastructure, execution, RCA, and reporting.

Add to Google preferred sources

Summarise with AI

Copied to Clipboard!
...

3000+ Browsers. One Platform.

See exactly how your site performs everywhere.

Try it free
...

Write Tests in Plain English with KaneAI

Create, debug, and evolve tests using natural language.

Try for free

Did you find this page helpful?

More Related Blogs

TestMu AI forEnterprise

Get access to solutions built on Enterprise
grade security, privacy, & compliance

  • Advanced access controls
  • Advanced data retention rules
  • Advanced Local Testing
  • Premium Support options
  • Early access to beta features
  • Private Slack Channel
  • Unlimited Manual Accessibility DevTools Tests