Android App Testing: Build a Device Matrix From Real Data

The Device Matrix Problem in Android App Testing

Summing every Android version from 13 downward in StatCounter's July 2026 worldwide Android version share gives 43.37% of devices. Most Android app testing plans are built from whatever handsets happen to be in the team's desk drawer.

This creates a specific failure mode. Development happens against the newest API level because Google Play requires it, while close to half the installed base runs versions that are three or four releases behind.

Teams already know fragmentation exists. What they lack is a repeatable way to turn it into a device list someone can sign off on, which is what the rest of this guide builds.

• Device choice is usually inherited from whoever set up the last test plan, then left alone.
• Coverage is expressed as a device count, so nobody can tell whether twelve devices is generous or negligent.
• The matrix rarely changes when a new Android version ships, so it drifts further from the installed base every quarter without anyone noticing.
• Emulator-only pipelines miss the OEM firmware behavior that produces a large share of field crashes.

Where the Real Android Distribution Data Lives

Before you can rank API levels you need a source, and the obvious one no longer works. The Android distribution dashboard no longer publishes a platform version breakdown. It now carries only Vulkan and OpenGL ES distribution tables, and its stated collection window ended on November 24, 2025.

The distribution figures that Android Studio shows in its New Project wizard come from a JSON file Google publishes separately. Fetching that file directly on August 7, 2026 returns this:

$ curl -sI https://dl.google.com/android/studio/metadata/distributions.json

HTTP/1.1 200 OK
Content-Length: 21101
Date: Fri, 07 Aug 2026 11:12:46 GMT
Last-Modified: Wed, 10 Dec 2025 23:26:48 GMT

$ curl -s https://dl.google.com/android/studio/metadata/distributions.json \
    | jq '[.[] | .apiLevel] | max'

36

The Android Studio distribution file was last modified in December 2025 and tops out at API level 36, but Android 17 shipped as API level 37 on June 16, 2026 per the Android Developers Blog. The distribution data a developer sees while choosing minSdk inside the IDE predates the current Android release and does not contain it.

That is workable as long as you know it.

In Play Console, the numbers you need are on the Statistics page: set the metric to active devices, then break it down by Android version and again by device model. The device catalog gives you the OEM rollup for the second axis.

Axis One: Ranking Android API Levels

This table combines the July 2026 worldwide version mix from StatCounter with the cumulative reach implied by Google's Android Studio distribution file.

The two right-hand columns answer different questions. Share tells you what to test. The minSdk reach column, derived from Google's per-API-level figures, tells you what you give up by refusing to support a level at all.

The Android 17 row shows the staleness problem in one line. It has real users at 0.35% in the StatCounter data, and no entry at all in the file Android Studio reads.

Android 13 is the row that catches teams out. It sits fourth by recency but third by share at 14.88%, ahead of Android 14.

A separate constraint comes from the store. Per Google Play target API level requirements, new apps and app updates must target Android 16 (API level 36) or higher from August 31, 2026, with an extension available to November 1, 2026, while existing apps must target API level 35 or higher to stay available to new users on newer devices.

That requirement governs what you compile against, not what you run on.

Axis Two: Ranking Android OEMs

Two devices on the same API level can behave differently, so the second axis uses worldwide mobile vendor market share for July 2026. StatCounter reports vendor share across all mobile devices, so the right-hand column rebases each figure against Android's 68.38% share of mobile in the same month to give its approximate share of Android devices.

One methodological note before you use this table. StatCounter's vendor dataset includes an Unknown bucket at 10.34% for July 2026, which would otherwise rank third. It is unidentified traffic rather than a manufacturer, so exclude it when ranking real OEMs and treat the rebased column as approximate for the same reason.

The Pixel row deserves attention for the opposite reason. At an estimated 2.4% of Android devices it is a rounding error in the installed base, yet it is the device most Android engineers develop and test on. Stock Android is the least representative build in the ecosystem.

Vendor skins matter because they change runtime behavior your app depends on:

• Aggressive battery optimization on several OEM skins terminates background work and delays notifications in ways stock Android does not.
• Permission dialogs and autostart managers differ per skin, so an onboarding flow verified on a Pixel can stall on a device with an extra vendor-level prompt.
• Default fonts, display density, and system gesture areas vary, which moves layout boundaries and changes what a screenshot test considers a regression.
• Regional mix reorders this whole table, so pull the same StatCounter vendor report filtered to each country that matters to you and weight the ranking by that country's share of your install base.

Screen geometry is a third axis. Per StatCounter screen resolution stats, the most common worldwide mobile resolution in July 2026 was 414x896 at 13.23%, followed by 360x800 at 9.14%, and density buckets vary independently of resolution, which this guide to DPI and screen resolution covers in depth.

Building the Matrix: P0, P1, and P2 Tiers

With both axes ranked, the matrix is their intersection, tiered by how much of the installed base each tier accounts for. The tiers below use worldwide July 2026 figures; substitute your own analytics wherever you have them.

• Rank Android versions by share and keep adding rows until you pass your coverage target. The top four versions reached 71.09% in July 2026.
• Rank OEMs by share in your priority regions, excluding the Unknown bucket, and keep the top three to five.
• Within each OEM, pick the model with the highest install count in your Play Console device catalog, not the newest flagship. A Galaxy A-series handset and a Galaxy S Ultra share a skin but differ on chipset, RAM, and thermal headroom.
• Pair the highest-share version with the highest-share OEM to form P0, and require every release to pass on that tier.
• Fill P1 with the next combinations by share, plus one deliberately old device that still sits above your minSdk floor.
• Put form factor outliers in P2, meaning a tablet, a foldable, and a low-RAM budget handset.
• Write the cumulative coverage percentage next to each tier, because that number is what justifies the device count.

Now do the coverage arithmetic, one axis at a time. A P0 of Android 16 and Android 15 covers 43.17% of the version axis (25.71 + 17.46), and Samsung plus Xiaomi covers an estimated 42.8% of the OEM axis once both are rebased against Android's share of mobile.

Report those two numbers separately and resist the temptation to multiply them. The axes are correlated, because newer Android versions are concentrated on newer and more expensive handsets, so 43.17% times 42.8% understates real cell coverage by an unknown margin. Two honest per-axis figures beat one invented product.

Adding Android 14 and 13 to P1 lifts version coverage to 71.09%, and adding Vivo lifts OEM coverage to roughly 51%. That is the shape of the curve: the first four devices buy most of the coverage, and each device after the eighth buys progressively less.

Tiering exists to protect feedback latency. P0 has to stay small enough that a pull request still returns a verdict in minutes, while P2 runs weekly and can afford to be broad. When you need month-over-month share movement to set your promotion threshold, Android device statistics tracks how these populations shift.

What Actually Breaks at Each API Level

A matrix is only worth running if each row can fail differently from the others.

Android 13 (API 33) introduced a runtime notification permission. Google's notification permission documentation states that on Android 13 or higher an app's notifications are off by default, and the app must wait to send notifications until the user grants POST_NOTIFICATIONS.

That single change splits a notification test into two paths, granted and denied, and only devices on API 33 or higher exercise it.

Android 15 (API 35) changed display and background rules for apps targeting it. Per Google's Android 15 behavior changes, apps targeting API 35 are edge-to-edge by default, and portions of the app may be obscured unless insets are handled.

• Data sync foreground services are limited to a total of 6 hours in a 24-hour period, after which the system calls the service's onTimeout method.
• BOOT_COMPLETED receivers can no longer launch dataSync, camera, mediaPlayback, phoneCall, mediaProjection, or microphone foreground services.
• TLS versions 1.0 and 1.1 are disallowed for apps targeting Android 15, which surfaces as a network failure rather than a UI defect.
• Requesting audio focus requires the app to be the top app or to be running a foreground service, otherwise the request returns AUDIOFOCUS_REQUEST_FAILED.

Android 17 (API 37) tightened large screen behavior. The Android Developers Blog announcement for Android 17 states that it removes the developer opt-out for orientation and resizability restrictions on large screen devices, which promotes tablets and foldables from optional to required.

Map each of these to the tier that covers it. Notification permission paths belong in P0 because they gate re-engagement, foreground service timeouts belong in P1 because they need longer runs, and large screen resizability belongs in P2 alongside the tablet and foldable.

Automated Android App Testing Across the Matrix

Automated Android app testing divides along one line before it divides along any other. Google's testing fundamentals guide states that the most important distinction for app developers is where tests run.

That line decides which parts of your matrix a test can even reach. Local tests never touch a device, so no cell of the matrix applies to them.

Google recommends running instrumented tests only in cases where you must test against the behavior of a real device. That recommendation is what keeps a twelve-device matrix affordable, because it caps how much work has to cross the device boundary at all.

Applied to the tiers, that gives a straightforward allocation:

• Business logic, view models, formatters, and parsing stay in local tests and never enter the matrix at all.
• The API-level behaviors from the previous section are instrumented by definition, because permission prompts, foreground service timeouts, and insets only exist on a device.
• P0 carries a thin instrumented smoke suite: launch, sign in, the primary conversion path, and the notification permission path.
• P1 carries the full instrumented regression suite, which is where the wall-clock cost lives and where parallel execution pays for itself.
• P2 carries layout and form-factor checks, which are the tests most sensitive to density buckets and folded versus unfolded state.

One myth to drop before you size any of this. Google publishes no numeric test-pyramid proportions. Its strategy guidance describes many small tests and relatively few big tests, and calls that a baseline that does not need to be followed strictly, so any precise split you see quoted is somebody's house rule presented as a standard.

Framework Fit: Espresso, UI Automator, and Appium

The matrix decides what you run on; the framework decides what you can write. Espresso, UI Automator, and Appium differ mainly on process boundary, which sets their limits.

Espresso's core statement chains a matcher, an action, and an assertion, and Google documents its API surface as ViewMatchers passed to onView, ViewActions passed to perform, and ViewAssertions passed to check.

// Espresso: matcher, action, assertion in one chain
onView(withId(R.id.name_field))
    .perform(typeText("Steve"))
    .check(matches(isDisplayed()));

Espresso deliberately blocks direct access to activities and views, so there is no getView or getCurrentActivity in the API. Its synchronization guarantee also means only one UI action or assertion occurs at any given time, which is why it is reliable but a poor fit for cross-app journeys. The Espresso tutorial covers that API surface in more depth.

A common split keeps Espresso for developer-owned tests that run on every commit against a small P0 tier, and uses Appium for the broader matrix where the same suite must cover Android and iOS. Both can execute against a device cloud: Espresso ships as an app APK plus a test APK bound to a device list, while Appium selects devices through capabilities.

Running the Matrix Without Buying the Devices

A matrix built from the tables above needs eight to twelve physical devices spanning several OEMs and four Android versions, kept charged, updated, and connected. That is a procurement and maintenance problem before it is a testing problem.

Three ways to solve it are worth weighing. A self-hosted device farm gives you full control and fixed hardware, at the cost of capital spend plus the ongoing work of OS updates and failed handsets. Firebase Test Lab is the Google-native option and integrates tightly with Android tooling, though its catalog leans toward stock Android and covers fewer OEM skins than the vendor mix above demands. A commercial real device cloud rents the OEM breadth directly.

TestMu AI's real device cloud covers the third path. It provides 10,000+ real Android and iOS devices spanning Samsung, Xiaomi, Oppo, Vivo, Motorola, OnePlus, and Google Pixel among 20+ manufacturers, with Android coverage from Android 9 through the current release and minor version variants included where behavior differs.

Four capabilities map onto specific rows of the matrix:

• Foldables including Galaxy Z Fold and Z Flip cover the P2 large screen row that Android 17 resizability changes now make mandatory.
• Per-session device logs, crash reports, and ANR logs isolate whether a failure is a genuine regression or specific to one OEM build, which is the daily triage question on a multi-device matrix.
• Network condition simulation exercises the low-bandwidth paths that budget devices in emerging markets actually experience.
• New flagship devices are added within weeks of market release, so P0 can track the devices your users are moving to.

Selecting a matrix cell in an automated run is a capability change, so switching cells needs no test code edit. The Appium with Java documentation shows the device and platform version capabilities in full.

// One matrix cell: OEM device + API level, selected by capability
DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability("deviceName", "Galaxy S20");
caps.setCapability("platformVersion", "11");
caps.setCapability("platformName", "Android");
caps.setCapability("isRealMobile", true);
caps.setCapability("build", "Android Device Matrix - P1");

One caveat if you also run Appium against your own hardware. The UiAutomator2 driver documents that appium:deviceName is not used to select the device under test and that appium:udid must always be set for parallel runs. A cloud grid resolves deviceName and platformVersion against its own catalog instead, so the same capability name behaves differently on a local Appium server.

Because each cell is just a capability set, the pipeline can iterate the matrix as a data structure. Running those cells concurrently is what keeps a twelve-device tier from adding twelve times the wall clock, and this walkthrough of Appium parallel testing covers that setup.

Refreshing the Matrix on Triggers

A matrix decays. StatCounter's worldwide Android version series shows Android 16 going from 0.81% in July 2025 to 25.71% in July 2026, passing Android 15 in March 2026 at 21.83% against 20.51%, so any matrix frozen in mid-2025 was materially wrong within nine months.

Four triggers are worth wiring up:

• A version in your install base crosses a share threshold you set in advance, for example 10%, which promotes it into P1.
• A Google Play target API level deadline approaches, which changes what you compile against and adds a new behavior-change test pass.
• Crash or ANR reports concentrate on a device or OS version that is absent from every tier, which is the clearest evidence the matrix has a hole.
• A flagship launches in a region that matters to you, or a supported version falls below the threshold and can be retired from P1 down to P2.

Retirement is the step teams skip, and it is why matrices only ever grow. Set the removal threshold when you set the addition threshold, and record the coverage percentage after each revision so the trend is visible.

Conclusion

Open Play Console, go to Statistics, set the metric to active devices, and break it down once by Android version and once by device model. Write down the top four versions and top three OEMs with their percentages. That table replaces the inherited device list and takes about ten minutes.

Then pair them into P0, P1, and P2 tiers, write the cumulative coverage for each axis next to each tier, and set the thresholds that will trigger the next revision.

To run those tiers without assembling a device lab, TestMu AI's app test automation platform executes Appium, Espresso, XCUITest, and Detox suites against real hardware, and the broader Android testing guide covers the test types that fill each cell once the matrix exists.