Computer Use Agents: How They Work and How to Test Them

What Is a Computer Use Agent?

A computer use agent is an AI system that drives software through screenshots and synthetic input. The agent reads a picture of the screen, picks a mouse or keyboard action, executes it, captures a fresh screenshot, and repeats until the task finishes or a step cap stops the run.

The defining property is the absence of an interface contract. There is no API to call, no selector to match, and no SDK to install against the target application. Anthropic states the position plainly: the application, not the model, captures the screenshot and performs every mouse and keyboard action.^[2]

That property is the whole trade. A screen-driven agent reaches legacy software with no integration surface and adapts when a button moves. It also inherits every weakness of vision: a misread pixel, a downscaled image, or a dense toolbar can put a click in the wrong place with no error raised.

How Do Computer Use Agents Work, Step by Step?

The loop is perception, reasoning, and action. A vision-language model reads the screenshot, returns a structured action such as a click at specific coordinates, your code executes that action on a real display, and a new screenshot restarts the cycle. The shape is identical across Anthropic, OpenAI, and Google.

• Your harness captures a screenshot of a display it controls, such as a virtual X11 display, a browser context, or a virtual machine, and sends it with the task text.
• The vision-language model reads the image plus the conversation history and returns a structured tool call naming an action and its target coordinates.
• Your harness translates that abstract action into a real event, such as a Playwright mouse click at the given x and y position.
• Your harness captures the new screen state and returns it as a tool result, and the model either requests another action or stops.

OpenAI documents the same cycle as five steps that terminate when the model stops returning a computer call, with nine action types covering click, double click, scroll, type, wait, keypress, drag, move, and screenshot.^[4] Stripped to its essentials, a harness is a short loop:

let screenshot = await capture(display);
let history = [{ role: "user", content: [task, screenshot] }];

for (let step = 0; step < MAX_STEPS; step++) {
  const reply = await model.act(history);
  const call = reply.toolCall;
  if (!call) break;                     // model believes the task is done

  await execute(display, call.action, call.coordinate);
  screenshot = await capture(display);  // observe the real result

  history.push(reply, { toolResult: screenshot });
  trace.append({ step, call, screenshot });  // keep every step, not the last three
}

The last line matters more than it looks. Anthropic recommends keeping only the last three screenshots and pruning every 25 turns to control cost, so the model sees a short window while your stored trace should retain all of it. The pruned screenshots are usually where a divergence started.

How Is Screen-Driven Automation Different From RPA and Playwright?

RPA and Playwright replay a fixed sequence against recorded selectors, so both break when the interface moves. A screen-driven agent re-derives every step from the current screenshot, which survives interface churn but makes the output probabilistic instead of repeatable. That single difference drives cost, speed, and every testing decision below.

The distinction is control logic, not the executor. Playwright is itself the execution layer inside several vendor reference harnesses. What changes is who decides the next step: an author who wrote it once, or a model deciding live on every run.

Read across the rows and the trade is clear: screen-driven agents buy reach and resilience with money, latency, and repeatability. The comparison in rpa vs ai covers where rule-based automation ends.

Should You Use a Desktop Agent or a Browser Agent?

Browser agents drive the DOM inside a single browser, and desktop agents drive the whole machine including native applications and the file system. Browser agents cost fewer tokens and avoid coordinate errors entirely. Desktop agents reach software that a browser cannot open.

A DOM-driven agent serializes page structure into text and asks the model to pick an element by role and name, so the framework resolves the click and grounding error disappears. That approach is blind to canvas, WebGL, PDF, and video, and to everything outside the page. Our roundup of browser agents covers that category.

In the Reddit thread "Is there a decent computer use model?" on r/LocalLLaMA, one commenter separated pixel-level control from accessibility-tree control, noting that open models remain weak at grounding and click accuracy on dense interfaces. The same commenter added that pixels are only needed for what the DOM cannot express, such as canvas or WebGL. Four other commenters converged on that same accessibility-tree route through playwright-MCP.

Which Computer Use Models and Platforms Can You Use Today?

Anthropic ships a beta computer use tool, OpenAI ships a built-in computer tool in the Responses API, Google ships Gemini Computer Use in preview, and Microsoft ships computer use inside Copilot Studio. Every one of those surfaces carries a beta or preview label as of August 2026.

Read those status labels as engineering constraints rather than marketing stages. A beta or preview surface carries no stability commitment, and OpenAI states that preview models can be retired with as little as two weeks of notice.^[7]

How Accurate Are Screen-Driven Agents Right Now?

Accuracy depends entirely on which benchmark is quoted. On OSWorld the human baseline is 72.36% and the first models managed 12.24%.^[9] On OSWorld 2.0, which uses long-horizon workflows, the best system finishes 20.6% of tasks end to end. Both figures describe the same generation of models.

The spread across those rows is the real finding. A number near 80% and a number near 20% can both be true of the same model, because they answer different questions on different task lengths. Microsoft reports the same split by surface, at roughly 80% success on web tasks against roughly 35% on desktop applications.^[12]

What Do OSWorld Leaderboard Scores Really Tell You?

An OSWorld leaderboard score is a single-run estimate. It tells you whether one attempt can succeed, and it says nothing about whether the same task succeeds twice. Repeated-execution numbers come in consistently lower, and the gap widens as the number of runs grows.

This is the gap between capability and reliability, and it is measured. Research on repeated execution states directly that reliability cannot be inferred from the ability to solve a task once and must instead be evaluated across repeated executions.^[13] Running 361 OSWorld tasks three times each produced a drop for every model tested.

Stretch the run count and the two metrics tell opposite stories. The same research reports Pass@10 near 78% while the agent succeeded on all ten executions for only about 36% of tasks. The distinction is the one tau-bench introduced: pass@k asks whether any attempt works, and pass^k asks whether every attempt works.^[14]

One derived observation is worth stating, and it is our reading of the published table rather than a quoted finding. If runs were independent, Pass^3 would equal Pass^1 cubed: Claude Sonnet 4.6 would land at 0.346 rather than 0.612, and GPT-5 at 0.191 rather than 0.454.

Observed reliability sits roughly two to two and a half times above that prediction, so outcomes cluster rather than scatter. Most tasks are near-deterministic passes or failures, and a smaller set is genuinely unstable. That justifies sorting your task set into consistently solved, inconsistently solved, and never solved, then treating only the middle bucket as flaky.

One reading habit protects you from a common misquote. OSWorld 2.0 scores tasks two ways, and Claude Opus 4.8 scores 54.8% on partial checkpoints against 20.6% on binary completion.^[1] Any figure in the 50% to 70% range on that benchmark is almost certainly checkpoint scoring, so ask which metric a chart shows.

How Much Does a Screen-Driven Agent Cost to Run, and Why Is It Slow?

Cost scales with screenshots, not with seats. Every step resends an image worth roughly 1,000 to 1,800 input tokens plus the growing history,^[2] and Microsoft Copilot Studio bills 5 Copilot Credits per step, or 15 on a premium model.^[3]

Per-step billing changes how you budget. Microsoft's worked example puts a four-step timesheet form fill at 20 credits on a standard model and 60 on a premium one, so an inefficient agent is directly more expensive. Anthropic adds a fixed 466 to 499 tokens of system prompt overhead before any pixels are sent.

Latency shares that root cause. Researchers behind OSWorld-Human describe current systems as practically unusable, citing end-to-end latency measured in tens of minutes for tasks humans finish in a few minutes. They also measured each successive step taking up to 3x longer than the early ones as the screenshot history grows.^[11]

Anthropic reaches the same conclusion and steers users toward work where speed is not critical, naming background information gathering and automated software testing as the fitting cases.

How Do Attackers Hijack a Screen-Driven Agent With Prompt Injection?

Indirect prompt injection hides instructions inside page text or inside the pixels of a screenshot, and the agent follows them. Anthropic measured a 23.6% attack success rate without safety mitigations and 11.2% with them, across 123 test cases and 29 attack scenarios.^[16]

The mechanism is structural rather than a bug. The agent receives operator instructions and page content through the same channel and cannot reliably tell them apart, so it follows both. Anthropic documents this directly, noting that instructions on webpages or inside images can override the operator's own.

Screenshots are an attack surface in their own right, not only the text on a page. VPI-Bench measured visual prompt injection across 306 test cases on five platforms and found agents deceived at rates up to 51% for screen-driven agents and up to 100% for browser agents on certain platforms.^[17] Vendors now run classifiers over the screenshot image itself.

Read the residual number as the design constraint. An 11.2% success rate after mitigations means roughly one in nine targeted attacks still lands.^[16] That rules out unattended agent access to payments, production data, or anything else where a single wrong action cannot be reversed.

How Do You Sandbox and Supervise a Screen-Driven Agent Safely?

Run the agent in a dedicated virtual machine or container with least privilege, allowlist the sites and applications it may touch, keep credentials in a vault rather than in the prompt, and require human approval before any irreversible action. Every major vendor documents these four controls.

• Dedicated machine: Microsoft recommends isolated machines used only for computer use, which removes cross-contamination from unrelated software and makes configuration auditable.
• Allowlist with a known gap: Access control restricts which sites and desktop apps the agent may act on, and Microsoft notes the model can still open a blocked site but cannot interact with it afterwards.
• Vaulted credentials: Copilot Studio stores passwords in Azure Key Vault or internal encrypted storage and injects them at the login prompt, so secrets never sit in the instruction text.
• Confirmation on risky actions: Anthropic runs injection classifiers over prompts and steers the model to ask for user confirmation when a screenshot looks like an injection attempt.
• Named human reviewer: Copilot Studio emails a specified reviewer when potentially harmful instructions are detected, and the run stops if nobody responds within the time limit.

Neither vendor presents its pause as a safety guarantee, and the honest reading is that these controls reduce blast radius rather than prevent hijacking. OpenAI states the requirement plainly: run computer use in an isolated browser or virtual machine, keep a human in the loop for high-impact actions, and treat page content as untrusted input.

Can a Screen-Driven Agent Run Your UI Tests?

A screen-driven agent can explore an interface, find bugs, and emit a Playwright script, but the deterministic script is what belongs in continuous integration. The agent is the explorer. The generated test is the artifact that runs on every commit.

One published evaluation shows both the promise and the limits. An engineering team at ML6 built three pipelines that generate user flows, hunt bugs, and write Playwright tests, and reported catching 27 of 28 functional bugs and 6 of 8 visual and polish bugs on a real application with implanted defects.^[18]

Carry the caveats with the number, because they are the point. That result is one team, one unnamed application, implanted bugs, and no disclosed run count, which makes it a single unreplicated evaluation rather than a measured rate. The same team notes that subtle visual regressions still slip through.

There is also a failure class that vision alone cannot catch. Practitioners on Hacker News described a React controlled select that renders the correct value after a click while the framework's internal state never updates. The screen looks right and the submitted payload is null.^[19] A vendor co-founder in the same thread conceded that a screenshot taken after an action does not solve it.

The rule that follows is to assert on the downstream outcome rather than the visual state. Check the request payload, the persisted record, or the confirmation identifier, and keep verification separate from execution so one perception error cannot both cause and hide a defect.

Turning an exploratory run into a maintainable suite is where most teams stall, because the flows an agent discovers still have to become tests someone owns. KaneAI generates test cases from natural-language intent, PRDs, or Jira tickets and exports them to Selenium, Playwright, Cypress, or Appium, so the deterministic script stays in your own repository.

• Multi-framework export: Generated tests export to Selenium, Playwright, Cypress, or Appium, so a team keeps its existing code ecosystem instead of adopting a new runtime.
• Intent-based authoring: Natural language, PRDs, Jira tickets, and recordings convert into executable cases, which covers the handoff from an exploratory finding to a committed test.

The getting started with KaneAI guide covers authoring and export in detail, and KaneAI lists the supported testing layers.

How Do You Test a Screen-Driven Agent Before You Ship It?

Run every task at least three times, report the all-runs-pass rate alongside the single-run rate, store a replayable step trace for each run, and gate on a three-valued verdict rather than a binary assertion. The five steps below turn that into a repeatable method.

Step 4: Triage the Failure Before Filing It

Ask whether it is a failure at all before asking what kind. An audit of 150 scored trajectories found 15.3% of FAIL verdicts were wrong, split into 10.7% evaluator false negatives and 4.7% broken tasks.^[20] Roughly one in seven bugs filed against the agent belongs to the harness.

Once a failure is genuine, the class matters because it decides who fixes it. Verification and feedback failures account for 39.3%, planning failures 35.2%, and execution or grounding errors only 13.9%.^[20] The single largest pattern is feedback-blind repetition at 29.5%, where the agent repeats an action that already failed because it never checked the result.

That distribution overturns the intuitive assumption. Most teams instrument for misclicks, but grounding is the smallest genuine category, and the dominant defect is an agent that does not look at what it just did. The canonical example is an agent clicking a results button 27 times while the screen never changed.

Two rules follow. Add a no-progress detector that aborts when consecutive screenshots are unchanged, and record the triage class on each run so the split is visible over time. The same audit reported only moderate annotator agreement on which tier a failure belongs to, so treat the class as a working label rather than a fact.

Which Workflows Fit Today, and Do Computer Use Agents Replace QA Engineers?

Strong fits today are repetitive workflows on systems with no API, background data gathering, and exploratory interface testing. Weak fits are high-volume, latency-sensitive, or irreversible transactions. Execution shifts toward agents, while oracle design, test strategy, and the release decision stay with human engineers.

The fit test is reversibility and tolerance for latency. Invoice processing, legacy data entry, and portfolio data extraction all tolerate a twenty-minute run that gets checked afterwards. A payment submission does not, because the residual injection rate and the silent-failure modes both land on an action nobody can undo.

The role question has a narrower answer than the headlines suggest. An agent that clicks through a happy path replaces the shallowest layer of manual checking. It does not produce the test strategy, the oracle, or the judgment about whether a release is safe, and someone still has to review what it produced.

In the Reddit thread "E2E QA with Codex 5.5 (allegedly)" on r/QualityAssurance, SDETs debated a fintech team's claim that a screen-driven agent handles its end-to-end QA. The top comment reported that Claude could not write meaningful E2E tests beyond a basic level in a 1.4 million line codebase, and another relayed the presenter's own phrase that the agent essentially clicks around. Most commenters rejected the claim.

The vendor claim at the centre of that discussion was never independently verified. That scepticism is the correct default for any single reported success rate in this category, including the ones in this article. For the scoring side, see testing non deterministic ai outputs and ai agent testing.

Conclusion

Start by running one real task three times and recording what changed between the runs. That single exercise tells you more about whether a computer use agent fits your workflow than any leaderboard row, because it measures the thing a pipeline actually needs: the same correct outcome, repeatedly.

The engineering position as of August 2026 is consistent across vendors. Capability is real and improving, every major surface is still labelled beta or preview, single-run scores overstate reliability, and verification failures rather than misclicks cause most genuine errors. Treat the agent as a system under test, keep a replayable trace, and let a deterministic script hold the merge gate.

The practical next step is to build the artifact before you build the agent, because a run you cannot reconstruct is a run you cannot debug. The same repeat-and-score method applies to conversational systems, and TestMu AI's AI agents cover the evaluation stages across those surfaces.

Sources and References Used

• [1] OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks - arXiv (June 2026)
• [2] Computer Use Tool Documentation - Anthropic (August 2026)
• [3] Automate Web and Desktop Apps with Computer Use - Microsoft Copilot Studio (August 2026)
• [4] Computer Use Tool Guide - OpenAI Platform Documentation (2026)
• [5] Computer Use in Azure AI Foundry Models - Microsoft (June 2026)
• [6] Computer Use with the Gemini API - Google (July 2026)
• [7] Deprecations and Model Retirements - OpenAI Platform Documentation (July 2026)
• [8] trycua/cua: Docker for Computer-Use Agents - GitHub (2026)
• [9] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments - XLANG Lab (2024)
• [10] ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use - arXiv (April 2025)
• [11] OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents - arXiv (June 2025)
• [12] FAQ for the Computer Use Tool - Microsoft Copilot Studio (May 2026)
• [13] On the Reliability of Computer Use Agents - Gonzalez-Pumariega, Agashe, Yang, Li and Wang, arXiv (April 2026)
• [14] tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains - arXiv (June 2024)
• [15] A Verification Layer for Browser Agents: Amazon Case Study - Hacker News Discussion (2026)
• [16] Piloting Claude in Chrome - Anthropic (August 2025)
• [17] VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents - arXiv (June 2025)
• [18] Computer Use Agents in Practice: Automating UI Testing - Violetta Nguyen, ML6 (July 2026)
• [19] Launch HN: Coasty, An API for Computer-Use Agents - Hacker News Discussion (2026)
• [20] How Benchmarks Mis-Score Computer-Use Agents - arXiv (July 2026)