Testing at Scale at Meta [Testμ 2023]

Prioritize Quality Over Quantity in Testing

Dmitry highlights the importance of choosing quality over quantity when it comes to software testing. He focuses on this principle, especially in tech giants like Meta, where it’s essential to ensure that each test conducted genuinely improves the software’s quality. This goes beyond running a lot of tests; it’s about making the software robust, reliable, and high-quality.

For a quick visual walkthrough of this concept, watch the video below from TestMu AI.

Subscribe to the TestMu AI YouTube channel for more tutorials on testing and automation.

He emphasizes that organizations, regardless of their size, face significant challenges related to testing quality. He uses the metaphor little kids, little problems; big kids, big problems to illustrate how minor issues in smaller organizations can become significant obstacles when dealing with larger enterprises. This narrative underscores the importance of maintaining high testing standards, no matter the organization’s size.

Striking the Right Balance in Testing

Dmitry introduces the idea of a Goldilocks test suite, which emphasizes finding the right balance between the quantity and quality of tests. This concept ensures effective testing—having just the right number of tests that strengthen software quality without overburdening the testing process with unnecessary complexities.

He encourages organizations to refocus on core use cases and critical test scenarios and avoid the common mistake of striving for 100% code coverage. This approach often leads to unnecessary complexities and increased maintenance costs.

Additionally, Dmitry highlights the importance of testing efficiency. By creating a suite that strikes the right balance between comprehensiveness and agility, organizations can speed up their testing processes while maintaining a strong commitment to quality, using Goldilocks test suite organizations can adopt a more practical and effective approach to software testing.

Role of Test Pyramid

Dmitry then explains a concept called the “Test Pyramid. ” It’s like building a solid foundation for a house. In this pyramid, there are three levels of tests, each serving a specific purpose.

At the bottom of the pyramid, we have unit tests. These are like the building blocks of testing. They’re small, fast to run, and check individual parts of the software. Unit tests are like checking if every brick in the house is sturdy and in the right place.

In the middle of the pyramid, we find integration tests. These tests ensure that different parts of the software work well together. It’s like making sure that the plumbing and electricity systems in the house are connected correctly and functioning as a whole.

At the top of the pyramid, we have end-to-end tests. These tests simulate how a real user interacts with the software. They check if everything works as expected from start to finish, just like testing if the entire house is comfortable and functional for someone living in it.

Dmitry’s point is that a balanced Test Pyramid is essential. It’s not about having a lot of one type of test and very few of another. Instead, it’s about having the right mix of unit, integration, and end-to-end tests. This approach helps maintain software quality efficiently, like building a house with a strong foundation while ensuring all its parts work together seamlessly.

Ice Cream Cone Problem

Dmitry explains a testing challenge known as the “ice cream cone” problem. It’s a situation where the Test Pyramid, which ideally has more unit tests at the base, fewer integration tests in the middle, and even fewer end-to-end tests at the top, gets inverted.

In the ice cream cone scenario, you have very few unit tests at the base. Instead, there are more integration tests, even more end-to-end tests, and a significant number of manual tests at the very top, like icing on an ice cream cone.

Dmitry points out that this situation can lead to several issues. First, it often means that unit tests, which are fast and provide quick feedback to developers, are neglected. Second, there is an over-reliance on manual testing, which is slower and less efficient. Third, it can result in duplicate efforts, with different teams writing the same end-to-end tests for the same functionalities.

The ice cream cone problem can impact testing efficiency, slow down development processes, and increase costs. Dmitry’s recommends to strive for a more balanced Test Pyramid, as it helps maintain quality while keeping testing manageable and cost-effective.

During his presentation, Dmitry also mentions the the book The Pragmatic Programmer – Explore It by Elisabeth Hendrickson. This book covers valuable resource for software developers related to software development, including aspects of testing and software quality.

Manual Testing is Not Core Testing

Dmitry then talks about manual testing and explains why it should not be the core or primary method of testing in a software development process. He points out several key reasons for this perspective:

• Manual testing relies on human testers to execute test cases, making it a time-consuming and costly process, particularly for large and complex software systems. This resource-intensive nature can slow down the development cycle.
• Manual testing may not cover all possible test scenarios due to human limitations. Testers might unintentionally overlook certain test cases or fail to replicate specific user interactions, leading to gaps in test coverage.
• Manual testing can be influenced by the tester’s bias, expectations, or experience. This subjectivity can result in inconsistent test results and missed defects that automated tests would catch consistently.
• Performing the same manual tests repeatedly can be monotonous and increase the risk of human error. Automated tests excel in executing repetitive tasks accurately and consistently.
• As the software evolves and grows in complexity, relying solely on manual testing becomes increasingly challenging and less practical. It can lead to bottlenecks in the testing process and hinder the ability to keep up with the pace of development.
• Heavy reliance on manual testing can lead to increased testing costs and project delays due to longer testing cycles, especially in large software projects.

Dmitry recommends using manual testing as a complementary approach alongside automated testing methods such as unit tests, integration tests and end-to-end tests. Automated testing can efficiently handle repetitive and comprehensive test scenarios, allowing manual testers to focus on exploratory testing, usability testing, and other qualitative aspects of the software—this balanced approach ensures efficient testing while maintaining software quality.

Why So Many UI Tests?

Dmitry advises against having an excessive number of UI tests due to several potential issues. Firstly, UI tests are generally slower and resource-intensive, causing longer testing times and build delays. Secondly, maintaining a large suite of UI tests can be complex and costly as software evolves, with UI elements changing over time, demanding continuous effort and resources.

Furthermore, more UI tests can lead to problems such as redundancy and duplication of effort among different teams. There is a common belief that more UI tests provide better coverage. Dmitry points out that having too many UI tests can slow down the feedback loop for developers.

Lastly, the increased maintenance costs associated with managing a large suite of UI tests should be carefully considered, prompting organizations to strike a balance between UI tests and other test types. This approach ensures efficient testing processes and meaningful coverage while minimizing the downsides of excessive UI testing.

How does Hourglass help?

Dmitry introduces hourglass in the context of software testing and team dynamics. In this concept, he highlights the importance of treating test code with the same level of attention and professionalism as production code. This means ensuring that test code is of high quality, follows established coding patterns and styles, and is subject to code reviews, just like production code.

He emphasizes the need for developers to be actively involved in reviewing test code and vice versa, promoting collaboration between developers and testers. Dmitry suggests a 1-to-1 model where for each story or feature, there should be at least one unit test or integration test accompanying it. Special care should be taken when reviewing critical or complex areas, such as login functionality, to prevent potential regressions that could affect multiple teams.

Dmitry mentions that having a dedicated “test champion” within the team can be helpful. This person’s role would involve consistently reviewing test code to maintain code quality standards. The overall goal is to bridge the gap between engineers and testers, encouraging shared responsibility for testing and fostering a culture of collaboration.

Additionally, Dmitry talks about the significance of testing as a crucial component of the feedback loop in software development. Testing isn’t just about finding defects; it plays a vital role in providing rapid feedback to developers. This feedback loop helps catch issues early, resulting in faster iterations and improved software quality.

Furthermore, he stresses the importance of team collaboration and shared understanding among different stakeholders, including engineers, testers, and project managers.

He mentions – transparent communication and alignment are key to addressing challenges in testing processes, particularly in large organizations—these principles act as a essential guiding principles for enhancing software testing practices.

Q&A Session!

• Could you provide an example of how Meta’s testing approach adapts to different project needs while maintaining consistent quality standards?
  Dmitry: There are several key aspects to how big organizations operate, and it’s not just a matter of chance. These organizations typically follow a structured approach, often starting with what I would describe as a bootcamp.

Regardless of the specific team you join, there is usually an initial onboarding process that resembles a bootcamp. For example, at Meta, where I work, this onboarding could last several weeks, during which you attend classes, lectures, workshops, and other training sessions. This phase aims to familiarize you with the organization’s products, tools, and coding practices. This includes learning programming languages like Rust, Swift, Objective-C, and adopting the coding style that the team follows.

However, big organizations also face the challenge of balancing freedom and consistency among their teams. While teams are generally allowed to choose their tools if they best suit the problem at hand, there are still certain standards that must be upheld. Within the same organization and team, you will typically find a set of standardized code styles.

For instance, if you are working with Java, you might be expected to adhere to Google’s coding style as a baseline. This ensures that even though teams have some flexibility, there is still uniformity in terms of coding conventions, such as indentations and code structure.

To provide a more general overview, the onboarding process, often likened to a bootcamp, serves as the initial step in understanding the organization’s practices. During this phase, you gain insights into how testing is conducted, what the code base looks like, and the overall architectural framework of the projects.

Importantly, big organizations tend to emphasize learning through practical experience rather than rigidly enforcing specific methodologies. They have expectations of their developers but allow room for individual creativity and problem-solving.

• How does Meta’s testing approach handle the diverse range of devices, platforms, and network conditions?
  Dmitry: In some organizations, you can use testing solutions to test across a variety of devices and browsers with different configurations. This includes manual and automated testing. Larger organizations often employ this approach, maintaining physical devices and collecting testing metrics.

The key is to focus on core use cases based on user metrics rather than testing everything. Prioritizing testing efforts helps allocate resources efficiently and avoid unnecessary testing on obscure configurations.

• How easy/difficult is it to work autonomously as a team at Meta? Is there a lot of red tape to go through to get things done?
  Dmitry: In a large organization, the experience can vary depending on the context. There’s a strong emphasis on autonomy and cross-functional collaboration. While certain factors like customer-facing work and data handling may introduce constraints, there’s always plenty of work to do.

Large organizations often operate as a network of startups within the larger entity, encouraging collaboration, and offering mobility between teams. This dynamic environment provides opportunities for growth and change if needed.

• Have you ever met a PM that was really difficult to get on board with these ideas/principles? How did you deal with it?
  Dmitry: Resolving conflicts within a team, particularly when differing with a Product Manager (PM) or higher-ups, involves maintaining an objective approach. It’s essential to give the benefit of the doubt and trust in the PM’s customer insights.

While standing firm on quality and technical excellence, valid arguments and compromises can be employed to reach common ground. Conflict resolution should prioritize the product’s best interests, considering the balance between speed and quality in fast-paced release cycles.

• Could you provide insights into how Meta’s testing strategy evolves as the company grows while still staying Agile?
  Dmitry: In the context of organizations like Meta and their evolution towards Agile methodologies and fast release cycles, it’s unlikely that organizations will revert to waterfall practices.

However, the concept of being Agile and flexible will continue to be important, and organizations will likely streamline their toolsets to improve efficiency. For smaller organizations looking to maintain a fast release cycle as they grow, a key best practice is periodically reviewing and deleting unnecessary code and tests.

Focus on writing tests that replicate core customer use cases rather than diving into edge cases first, ensuring that development aligns with customer needs to stay Agile and efficient.

Have more questions? Shoot them here at the TestMu AI Community.

Tracking tests across tools and teams? TestMu AI's Test Manager keeps test cases, runs and results in one place.

To run your first test, start free, or

book a demo

for a full walkthrough of the product.