Mobile Testing Pyramid For Agile Teams

Focus Areas of the Mobile Testing Pyramid

The following section explains the focus areas of the Mobile Testing Pyramid. The section is arranged in the bottom-to-top order for the testing pyramid.

Desktop browsers

The bottom-most section of the testing pyramid states “browsers,” corresponding to testing the mobile app on the desktop browsers. Since it is the broadest section, it has the maximum number of tests.

Mobile testing on desktop browsers is not a hard job. Just run the web app on the browsers, rescale them for responsive design test and UI glitches, and you are good to go. It is the best choice to start with when dealing with hybrid applications or pure web apps.

Read: Difference Between Web vs Hybrid vs Native Apps

Shown below is an example of resizing a web app on a desktop browser.

Advantages

• Extremely fast: Using desktop browsers for testing is a speedy job. Using the best automation testing frameworks like Selenium, you can run thousands of parallel test cases and use the headless approach. Refer to our page to know what is Selenium?
• Scalable: As mentioned in the previous point, you can use Parallel testing to cut down test execution time by leveraging a mobile device testing cloud like TestMu AI. New browsers are also easy to acquire as they are available readily.
• Easy for cross-platform: Desktop browsers are easy to work with because you don’t need to install a completely new OS as you would do on a native app. You can use cloud-based cross browser testing platforms to launch browsers from a single platform.
• Feasible and Easily available: Desktop browsers are readily available and are free to use. So you need very little investment to start with.

Disadvantages

• Whatever you do, you are still on a desktop: No matter how rigorously you test on mobile on the desktop browsers, the fact that you are using desktop resources won’t change. This affects how your results appear on the reports and the app’s performance.
• No “mobile-like” behavior: When you use a desktop browser to test an application. There are no network variations, including blockages, no incoming calls in between, no sudden freezing of devices due to lack of resources etc. These things are necessary when performing app testing on mobile as these are normal mobile behavior. Any crash due to these can affect your viewership.
• “Just not a device”: A self-explanatory sentence as said by Kwo Ding: “this is just not a device”.

Also, Read: Best Mobile App Testing Frameworks For Android and iOS Apps

The above-discussed analyses are true. Still, since browsers secure a place in the Mobile Testing Pyramid, they are unique and need to be considered in the testing.

The following focus areas are important while testing mobile apps on desktop browsers.

• Functional System testingDesktop browser tests efficiently perform functional system testing, which could be considered a primary testing method.
• Responsive designOne of the most important outputs of desktop browser tests for mobile apps is to check its responsive web design. Responsiveness is an important characteristic and should be applied in every application you build using a browser.

Also Read: Why Responsive Web Design Is Important?

The following image reflects a responsive web app rendered across two different sized screen devices on LT Browser.

Here is the quick rundown of LT Browser.

• Cross Browser TestingCross browser testing ensures the cross browser compatibility property of your mobile app across numerous browsers and operating systems.
• Overall Visual layoutPerforming Visual regression testing on desktop browsers can also help verify the overall visual layout of the mobile app. Kwo Ding ensures to focus on the word “overall” here. This is because desktop browsers cannot be trusted to deliver the accurate UI layout that you would get on a mobile screen. You can get a big picture but not the specifics or minute nuances of the UI elements.

Read More: UI Testing: What It Is and How Should You Do It

Emulators and Simulators

After reading the desktop browser section, anyone can be convinced that they are probably an excellent way to go through and launch our app without caring too much. If you think of the same question, let’s test an Android app on a mobile app emulator.

Output

Shown below is the mobile web tested application. It shows glitches in navigation when rendered on a mobile emulator. Such occurrences can happen with you, too, if you rely on desktop browser testing entirely for the mobile app.

Testing Mobile apps on Emulators and Simulators is more inclined towards mobile devices’ reality (especially the emulator) than desktop browsers. This is because emulators consider hardware while rendering a mobile application or a mobile web. Recent emulators can also mimic certain functionalities of mobile devices like incoming calls and network variation. This provides a better idea of our mobile app execution for the end-user.

Also, Read: Why Android Application Testing Is Important?

Since it is the next segment to browsers, it should be performed after you are convinced with desktop browser tests. If an error is found in a desktop browser, it will reflect different processes. However, its vice versa is not true.

Advantages

• Quick to use: The emulators are straightforward to set up, and you can quickly start testing.
• Scalable: Emulators are easily scalable as you just need to install a new device when it is launched. You don’t need to worry about procuring one.
• Bring a few native functionalities: Emulators bring a few native functionalities into the testing environment, such as receiving calls and messages.
• Debugging facilities: Tests run on emulators are easy to debug as you use a desktop screen to work. In addition, the logs can be saved and analyzed quickly.
• Fast to execute: Emulators are faster to execute the test as you just upload a native binary file or enter a URL to see quick results. You can also use parallelization in emulators to execute tests faster.
• Gives you a head-start: The emulators are available as soon as the device is announced. Sometimes it is available based on rumoured specifications too. So testing on emulators can give you a head start to those waiting to receive the real device when it comes to the market.

Disadvantages

• Can’t imitate each device: Emulators cannot mimic each device in the market. You would find a lot of skins for many devices, but they run on a fixed stock underneath.
• Inaccurate resource usage: Emulators do not accurately picture the resource usage such as CPU, GPU, memory, or others. They provide the metrics, but that is hardly comparable due to the absence of real hardware.
• Absence of native mobile behavior: Emulators are better than desktop browsers in bringing a few mobile-like behaviors to the testing environment. However, it cannot mimic a lot of native mobile device APIs such as camera behavior, Bluetooth behavior, other wireless connections such as NFC etc.
• ARM-based emulators are slow: Kwo Ding mentions that almost all Android devices work on the ARM architecture. But, emulators working on ARM architecture are slower in execution. Therefore, they do not give a precise output from the tests.

Practically, emulators play a vital role in Mobile testing, and therefore, a tester should focus on the following areas while working with the emulators:

• Functional end-user flowsFunctional end-user flows determine whether your application works from path A to path B when navigated like an end-user. Again, emulators are an excellent choice to confirm that.
• Native API integrationA tester can focus on native API integration available in the emulator to verify their correctness with respect to the application. These may include call features, message features, GPS features etc.
• Visuals testingA tester can check the visuals but not go too deep into a device-specific UI. The high-level visuals can be verified here.
• Touch interactionsEmulators are an excellent primary choice to test for touch interactions of the mobile application. Even though it is not as accurate as a real device, it can give an idea about the high-level interaction and mobile app performance.

Real device

Kwo Ding describes the real device section as “the real thing” in his introduction to the Mobile Testing Pyramid. He is right in this statement. The benefit of real devices is that you can test crucial mobile app testing scenarios in real-user conditions.

Read More: Real Device Cloud.

Looking to perform Android and iOS app testing on Real Device Cloud, check out our video below:

But that does not mean they lack any downside. Let’s compare its pros and cons.

Advantages

• All native APIs: A real device has all the native APIs and can be used along with the Mobile app testing.
• Real conditions: A real device presents real conditions to the tester that an actual end-user would face. This gives an accurate idea of the behavior of our mobile application.
• Faster ARM (Android) tests: As discussed earlier, ARM-based emulators are slow in test execution which majorly corresponds to Android devices. Therefore, real device test execution is much faster in Android.
• Everything is real: Ding does not let it slip again that the fact here is that they are real devices, and hence every metric here is accurate. So you get real CPU usage, real battery metric, real GPU usage, real logs etc.

Disadvantages

• Huge costs- Real devices are not easy to acquire as an emulator. They are too costly and can be a significant part of your overall testing budget. The current average cost of one device globally is close to USD 300. Acquiring 100 devices can put USD 30,000 just for procurement which can be a big deal for startups.
• Heavy maintenance: Another major factor in real device testing is the continuous maintenance it asks for from a user. A significant factor here is the continuous release of mobile phones. In 2021, 47 new mobile devices were released worldwide, a below-average number if you follow over the years. As a tester, you need to keep acquiring new devices to raise costs.

Except for cost and maintenance, real devices hardly bring any disadvantages to the table. But, I have a solution to eradicate this anomaly too. Why not try cloud-based real device labs that give you real devices for testing without actually purchasing even a single device.

TestMu AI platform offers a real-device lab with top-notch features integrated into it. With 3000+ screens to test on, TestMu AI real devices provide accurate metrics, device usage and logs in their detailed performance report. Along with it, you get network throttling, screen recording, sharing options and a debugger integrated into the platform. Once done, you can make use of tons of TestMu AI integrations that can facilitate your testing from a single place.

In the Mobile Testing Pyramid image below, notice that the real device section is distributed to two-segments instead of one, like emulators and browsers.

The lower segment containing the word “devices” denotes the usability or functional testing part of the real devices. The upper segment is for the non-functional testing part.

These will also reflect in the following focus area of real devices.

• Usability

  The usability of the mobile application can be tested thoroughly and most efficiently in a real device. This may include touch interaction connections such as Bluetooth, NFC and more.

• CPU

  Focus on CPU usage and the metrics it generates as they will be accurate according to the device in use.

• Visuals

  Kwo Ding refers to this section as the “leftover visuals” you could not see in other testing (segments). Desktop browsers and emulators can clear out most of the visual stuff for your application. But for confirmation, you can focus on this here with lower priority.

• Manufacturer’s sauce

  A few manufacturers put up their UI environment on Android. For example, Samsung has its own One UI that comes on top of Android’s UI engine. This could affect your mobile application UI and therefore is a good focus area for a tester.

• Specific built-in browsers

  A few device manufacturers also release their devices with specific built-in browsers, such as Samsung with Samsung Internet. They may contain a tiny percentage of users, but it is always a good practice to account for everyone and every scenario before releasing the mobile app.

  This was all about the Mobile Testing Pyramid and the detailed exploration of its various segments. The Mobile Testing Pyramid is an excellent reference for structured testing and good performance. A brief analysis can be therefore dictated here for the agile teams.