Video Quality Testing: From VMAF Scores to Playback QoE

A team ships a new encoding ladder. The dashboard shows the reference score went up on every rendition, the release notes say quality improved, and within a day support is fielding complaints that videos take forever to start on older Android phones.

Both things are true at once. The encode did get better and the viewers did get a worse experience, because the two were never measuring the same thing.

Video quality testing measures both how good a video looks compared to its source and how reliably it reaches the person watching. Those are different questions with different tooling, and most confusion in this area comes from a team measuring one and making claims about the other.

The delivery half carries the sharper business consequence. In a large-scale study of viewer behaviour presented at the ACM Internet Measurement Conference, Krishnan and Sitaraman found that viewers begin to abandon a video if it does not start within two seconds, and beyond that point every additional second of delay produced roughly a 5.8 percent increase in the abandonment rate, as reported by ScienceDaily.

No encoder setting fixes a two-second startup. That is a delivery problem, and it is invisible to every metric that looks only at pixels.

Keeping the families apart is most of the discipline.

• Reference-based scores - run offline against a file, deterministic, repeatable, and cheap. They answer whether this encode is a faithful version of that source.
• Delivery metrics - collected from a real playback session on a real network. They answer whether a viewer got a watchable stream, and they cannot be computed from a file alone.

The audio equivalent splits the same way, and the reasoning transfers almost line for line if you also ship voice, as covered in voice quality testing.

Three metrics dominate, and they disagree because they were built to measure different things.

Running one is unglamorous, which is the point: the same clip and the same command give the same number every time, so it belongs in the pipeline rather than in someone's judgement.

# score an encode against its source
ffmpeg -i encoded.mp4 -i source.y4m \
  -lavfi libvmaf=log_path=vmaf.json:log_fmt=json \
  -f null -

# gate the build on a regression, not on an absolute number
#   fail if the mean drops more than 2 points below the last release
python3 compare_vmaf.py vmaf.json baseline.json --max-drop 2

Note the second command. Gating on an absolute score invites arguments about what number is good; gating on a regression against the previous release asks a question everyone can answer.

Note: Encode scores run anywhere. Playback verdicts need hardware, and TestMu AI provides 10,000+ real devices with network throttling to measure it. Try it free.

These come from sessions, not files, and they are the ones your viewers actually feel.

Measure each under constrained network conditions rather than office wifi, because the failures worth catching happen on a congested mobile connection and never reproduce on a fast one. For the delivery pipeline itself, including adaptive bitrate protocols and DRM, the OTT testing guide goes deeper than this article does.

The disagreements are where the useful bugs live, and there are three common shapes.

• Better encode, worse experience - a higher bitrate ladder raises the reference score and pushes start time past the point where people leave. This is the release in the opening of this article.
• Same score, different device - an encode that plays smoothly on a recent phone stutters on an older one whose decoder handles the codec in software rather than in hardware.
• Good average, bad session - averaged metrics hide the segment of users on a specific device or network for whom playback fails outright, which is why you segment before you aggregate.

Whenever the two families disagree, believe the session. A viewer never sees your reference score.

Set it up in the order that gets you a verdict soonest.

• Pick a small set of source clips that stress different things: fast motion, dark scenes, fine detail, and a static talking head. Four clips of ten seconds each is enough to start and cheap enough to run often.
• Score every build against the previous release rather than against an absolute target, and store the baseline alongside the code that produced it.
• Define the delivery budget before you measure: the start time you will accept on your slowest supported network, and the rebuffering ratio you treat as a failure.
• Run playback on real devices across your supported matrix, with the network throttled to the conditions your actual users have rather than the ones your office has.
• Segment results by device and network class before averaging, since the whole point is to find the group being failed.
• Keep the session recordings for anything that fails, because a stutter is far easier to argue about with the footage in front of you.

Codec support is where playback most often fails outright rather than degrading, and it varies by browser and by device generation in ways that are easy to assume away.

Before you build a matrix, check what your target browsers actually decode: the current state of AV1 browser support and of HEVC compatible browsers both differ enough between engines to change which renditions you need to ship.

An emulator will not settle any of this. It does not carry the hardware decoder that determines whether a codec plays smoothly, drains the battery, or falls back to software and stutters, so encode scoring can run anywhere while playback verdicts need the real thing.

The delivery half of this work needs hardware and controlled conditions, which is what the real device cloud provides.

• Real hardware decoders - testing runs on 10,000+ real devices rather than emulated ones, so codec support and decode behaviour are the device's own rather than a simulation of it.
• Network conditions you choose - bandwidth caps, latency injection, and packet loss percentages let you reproduce the congested connection where rebuffering actually happens.
• Recordings by default - every session is recorded at the device's native resolution and available immediately afterwards, which turns a disputed stutter into something you can watch.
• Camera input you control - for flows that capture video rather than play it, video injection feeds a known file into the camera stream so the input is identical on every run.

Reference scoring stays where it belongs, in your own pipeline next to the encoder, since it needs no hardware and benefits from running on every build.

Write down two numbers this week: the reference-score drop you would refuse to ship, and the start time you would refuse to ship on your slowest supported network. Those two thresholds turn a quality conversation into a gate.

Then measure both on every release, segment before you average, and when the encode score and the session disagree, trust the session. If your product also puts an AI agent on camera, the quality bar shifts again and is covered in video simulation testing.

To measure playback on real hardware, create a free TestMu AI account, which includes 60 minutes of real device and browser testing with no credit card required.