How to Run Gemini Computer Use on a Real Cloud Browser

What Is Gemini Computer Use?

Gemini Computer Use is a capability within the Gemini API that enables a model to operate a user interface by looking at screenshots and suggesting actions. As per the Google docs as of November 2026, it's a tool to "build browser, mobile, and desktop control agents" with a separate environment value and action set for each (browser, mobile and desktop). It runs on the Interactions API, and the docs' code samples use gemini-4-hydrogen (Gemini API docs).

That is wider than the original release. Google introduced the Gemini 2.5 Computer Use model on October 7, 2025, tuned mainly for web browsers (Google blog).

Google's docs also tell you to prompt the end user before any action the model flags with require_confirmation, and the code below does that.

This guide covers the browser environment. Gemini Computer Use does not include a browser: your client code, typically Playwright, executes every action. That is where a cloud browser comes in.

For the general theory of screen-driven agents, see our explainer on computer use agents.

How Does the Gemini Computer Use Loop Work?

The Gemini Computer Use loop has five steps, and your code owns three of them:

• Send the goal and a screenshot. The first request will contain a text item with the task and an image item with a base64 PNG screenshot. It will also contain the computer_use tool with environment: "browser". Google's request example also includes the opt-in enable_prompt_injection_detection: true, which will check each screenshot for any hidden adversarial instructions and will block execution if any are found.
• Read the proposed actions. The steps in the response's steps list are function_call steps, each with a name (e.g., click), arguments, and an id. One turn can contain multiple calls.
• Execute each action in the browser. That's your code.
• Return a result for every call. Each result is a function_result with the call's name and call_id, a text item holding JSON with the current url, and a new PNG screenshot.
• Continue by passing the previous_interaction_id. Stop when the model makes no further function calls (i.e., when it returns nothing) or when your turn limit is reached.

Coordinates come in on a normalised 0-999 grid so the executor has to scale them to real pixels (x / 1000 * width). Google's examples are all on a 1440x900 screen.

Why Run Gemini Computer Use on a Cloud Browser Rather Than Locally?

A local Playwright browser is perfectly fine for a first Gemini experiment. Three things change when a Gemini agent actually runs:

• Clean, fixed-size screenshots. Gemini makes its decision on the pixels, so every run requires a fresh browser at the size that the executor scales to (1440x900 in this case) with no cookies, banners or pop-ups from the previous task remaining.
• A replay of what the model saw. If Gemini has misclicked, you need the video and the logs from that session to find the screenshot that caused it to go off the rails.
• Go into the private apps. Gemini agents that are acting on staging or internal tools require a tunnel rather than a public IP.

Google's reference implementation (google-gemini/computer-use-preview) has two environments as of Nov 2026: playwright (runs the browser locally) and browserbase (connects to a hosted Browserbase browser - GitHub).

TestMu AI Browser Cloud is the product for this. Each test gets you an isolated real Chrome browser in the cloud, a built-in tunnel to staging and localhost, and a full session replay. It plugs into the same Playwright executor via its Playwright adapter (as per the diagram below), and TestMu AI has a Gemini + Browser Cloud cookbook up on GitHub.

If you're looking at the various hosted browser options, we've got a side-by-side Browserbase vs TestMu Browser Cloud comparison covering the pricing, session limits, stealth and SDKs.

Browserbase setup facts are from Google's reference repository (read 12th November 2026). Browserbase tunnel and replay facts are from our own comparison (read Browserbase docs 5th November 2026). TestMu AI facts are from the Browser Cloud product page and docs.

Note: Can you build a browser agent on Gemini? Install the Browser Cloud SDK via npm i @testmuai/browser-cloud and point the agent below at your own task.

How Do You Connect Gemini Computer Use to Browser Cloud?

You'll need Node.js 18 or later (the Browser Cloud Playwright adapter requires it), a Gemini API key, and TestMu AI credentials. Install the two SDKs:

npm i @testmuai/browser-cloud @google/genai

Then set four environment variables from your secrets manager or your CI secrets (never in code or in URLs): LT_USERNAME and LT_ACCESS_KEY (your TestMu AI credentials), GEMINI_API_KEY and GEMINI_MODEL (a model that's on the Google docs list as being one that supports Computer Use).

The script below creates a Browser Cloud session with the Playwright adapter, runs the Gemini Computer Use loop, asks a human for approval before any action that Gemini flags, and then always releases the session. The task and start URL are both command-line parameters, so no test data is hardcoded.

// agent.mjs: Gemini Computer Use + TestMu AI Browser Cloud (Node.js 18+)
// Usage: node agent.mjs "<task>" "<start-url>"
import { GoogleGenAI } from '@google/genai';
import { Browser } from '@testmuai/browser-cloud';
import readline from 'node:readline/promises';

const [task, startUrl] = process.argv.slice(2);
if (!task || !startUrl) throw new Error('Usage: node agent.mjs "<task>" "<start-url>"');

const WIDTH = 1440, HEIGHT = 900;               // screen size used in Google's examples
const MODEL = process.env.GEMINI_MODEL;          // e.g. gemini-4-hydrogen; see the docs' model list
if (!MODEL) throw new Error('Set GEMINI_MODEL to a model that supports Computer Use');
const MAX_TURNS = Number(process.env.MAX_TURNS ?? 20);
const TOOLS = [{ type: 'computer_use', environment: 'browser', enable_prompt_injection_detection: true }];

const ai = new GoogleGenAI();                    // as in Google's JS examples; set GEMINI_API_KEY in the env
const client = new Browser();

const session = await client.sessions.create({
  adapter: 'playwright',
  dimensions: { width: WIDTH, height: HEIGHT },
  lambdatestOptions: {
    build: 'gemini-computer-use',
    name: task.slice(0, 60),
    'LT:Options': {
      username: process.env.LT_USERNAME,
      accessKey: process.env.LT_ACCESS_KEY,
    },
  },
});
const { browser, page } = await client.playwright.connect(session)
  .catch(async (err) => { await client.sessions.release(session.id); throw err; });  // release on a failed connect
console.log('Session dashboard:', session.sessionViewerUrl);

const px = (v, size) => Math.round((v / 1000) * size);   // 0-999 grid -> pixels
const screenshot = async () => (await page.screenshot({ type: 'png' })).toString('base64');

async function execute(name, a) {
  switch (name) {
    case 'click':        await page.mouse.click(px(a.x, WIDTH), px(a.y, HEIGHT)); break;
    case 'double_click': await page.mouse.dblclick(px(a.x, WIDTH), px(a.y, HEIGHT)); break;
    case 'triple_click': await page.mouse.click(px(a.x, WIDTH), px(a.y, HEIGHT), { clickCount: 3 }); break;
    case 'right_click':  await page.mouse.click(px(a.x, WIDTH), px(a.y, HEIGHT), { button: 'right' }); break;
    case 'middle_click': await page.mouse.click(px(a.x, WIDTH), px(a.y, HEIGHT), { button: 'middle' }); break;
    case 'move':         await page.mouse.move(px(a.x, WIDTH), px(a.y, HEIGHT)); break;
    case 'mouse_down':   await page.mouse.move(px(a.x, WIDTH), px(a.y, HEIGHT)); await page.mouse.down(); break;
    case 'mouse_up':     await page.mouse.move(px(a.x, WIDTH), px(a.y, HEIGHT)); await page.mouse.up(); break;
    case 'drag_and_drop':
      await page.mouse.move(px(a.start_x, WIDTH), px(a.start_y, HEIGHT));
      await page.mouse.down();
      await page.mouse.move(px(a.end_x, WIDTH), px(a.end_y, HEIGHT));
      await page.mouse.up();
      break;
    case 'type':
      await page.keyboard.type(a.text);
      if (a.press_enter) await page.keyboard.press('Enter');
      break;
    case 'press_key':    await page.keyboard.press(a.key); break;
    case 'hotkey':       await page.keyboard.press(a.keys.join('+')); break;
    case 'key_down':     await page.keyboard.down(a.key); break;
    case 'key_up':       await page.keyboard.up(a.key); break;
    case 'scroll': {
      const d = a.magnitude_in_pixels ?? 300;
      await page.mouse.move(px(a.x, WIDTH), px(a.y, HEIGHT));
      await page.mouse.wheel(
        a.direction === 'left' ? -d : a.direction === 'right' ? d : 0,
        a.direction === 'up' ? -d : a.direction === 'down' ? d : 0,
      );
      break;
    }
    case 'navigate':     await page.goto(a.url); break;
    case 'go_back':      await page.goBack(); break;
    case 'go_forward':   await page.goForward(); break;
    case 'wait':         await page.waitForTimeout((a.seconds ?? 1) * 1000); break;
    case 'take_screenshot': break;                // the screenshot is attached below anyway
    default: return { error: `Action not implemented in this executor: ${name}` };
  }
  return {};
}

async function confirm(explanation) {
  if (!process.stdin.isTTY) return false;         // no human present (e.g. CI): decline
  const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
  const answer = await rl.question(`Gemini asks for confirmation: ${explanation}\nAllow? (y/N) `);
  rl.close();
  return answer.trim().toLowerCase() === 'y';
}

try {
  await page.setViewportSize({ width: WIDTH, height: HEIGHT });
  await page.goto(startUrl);
  let interaction = await ai.interactions.create({
    model: MODEL,
    tools: TOOLS,
    input: [{
      type: 'user_input',
      content: [
        { type: 'text', text: task },
        { type: 'image', data: await screenshot(), mime_type: 'image/png' },
      ],
    }],
  });

  for (let turn = 0; turn < MAX_TURNS; turn++) {
    const calls = (interaction.steps || []).filter((s) => s.type === 'function_call');
    if (calls.length === 0) break;                // the model has finished

    const results = [];
    for (const call of calls) {
      const args = call.arguments ?? {};
      let ack = {};
      if (args.safety_decision?.decision === 'require_confirmation') {
        if (!(await confirm(args.safety_decision.explanation))) {
          console.log('Action declined by the user. Stopping.');
          process.exitCode = 1;
          break;
        }
        ack = { safety_acknowledgement: true };
      }
      let outcome;
      try { outcome = await execute(call.name, args); }
      catch (err) { outcome = { error: String(err.message ?? err) }; }
      // let the page settle before the screenshot, as Google's sample does
      await page.waitForLoadState('load', { timeout: 5000 }).catch(() => {});
      await page.waitForTimeout(1000);
      results.push({
        type: 'function_result',
        name: call.name,
        call_id: call.id,
        result: [
          { type: 'text', text: JSON.stringify({ url: page.url(), ...outcome, ...ack }) },
          { type: 'image', data: await screenshot(), mime_type: 'image/png' },
        ],
      });
    }
    if (process.exitCode === 1) break;

    interaction = await ai.interactions.create({
      model: MODEL,
      tools: TOOLS,
      previous_interaction_id: interaction.id,
      input: results,
    });
  }

  for (const step of (interaction.steps || []).filter((s) => s.type === 'model_output')) {
    for (const block of step.content ?? []) if (block.type === 'text') console.log(block.text);
  }
} finally {
  await browser.close();
  await client.sessions.release(session.id);
}

How Do You Debug a Gemini Computer Use Run?

The vast majority of failed computer use runs are due to one bad screenshot read: a cookie banner covering the button, a page that hadn't quite finished loading or a click that was scaled to the wrong pixel. Debugging is all about seeing precisely what the model saw.

The SCOPE checklist for Gemini Computer Use runs:

• Screenshot size matches the dimensions (1440 x 900 in this case). If it's a different size, then every click is going to be off.
• Confirmation path is tested. Run a task that causes require_confirmation to be triggered and verify the agent stops when you decline. Keep enable_prompt_injection_detection on for any pages you don't control.
• Observability is on. Open session.sessionViewerUrl to view the session's dashboard page and then replay the video, console and network logs on the Web Automation Dashboard (debugging docs).
• Page load waits are dealt with. If a click occurs on a half-loaded page, then either add a wait before the screenshot or have the model call wait.
• End conditions are set. Use a turn limit (MAX_TURNS), a session timeout and a clear 'done' state in the task text.

Browser Cloud records video, console logs, network logs and agent commands all on one timeline for every session. We'll explain how we built session recording into Browser Cloud.

For a quick visual walkthrough of this concept, watch the video below from TestMu AI.

Subscribe to the TestMu AI YouTube channel for more tutorials on testing and automation.

What Happens If You Port a Gemini Executor Over to OpenAI or Claude?

All three follow the same screenshot -> action -> screenshot loop, and all three hand execution over to your code. The differences are in three things that matter when you're porting an executor: the coordinate system, how the tool is declared, and where the confirmation signal is coming from. As of 12th November 2026, from each vendor's computer use docs:

Sources:

• Gemini API docs
• OpenAI computer use guide
• Claude computer use tool docs

The practical difference for a cloud browser is the coordinate step. A Gemini executor has to scale all coordinates from the 0-999 grid, whereas the OpenAI and Claude executors just use pixels directly as long as the screenshot isn't resized.

The browser session itself is the same. It's the same Browser Cloud session and the same Playwright page behind any of the three loops.

What Should You Check Before Shipping a Gemini Browser Agent?

Before a Gemini browser agent acts for real users, check:

• Scope: the agent is only able to get to the sites that the task requires. Use a dedicated test account, never a personal one.
• Human in the loop for anything that is flagged as require_confirmation (i.e., payments, account creation, legal terms, etc.).
• Replays are kept for every production run so you can audit exactly what the agent did.
• Regression runs: same task, same page state, same end result. Replay the failures before you go blaming the model.
• Cost and time constraints: turn limits and session timeouts are set in line with the task at hand.

Conclusion

Gemini Computer Use gives you the decisions. You still need a reliable place to carry them out. Keep the loop small: screenshot in, actions out, a result for every call, a human for anything that gets flagged.

Then you move execution from the laptop to isolated cloud sessions that you can replay. That final step turns a demo into an agent that you can run, debug and trust.

Try it on your own task

Note: Sign up for Browser Cloud, run agent.mjs against a page you own, and watch the replay in the dashboard.

Giving an AI agent a browser? TestMu AI's Browser Cloud runs each task in a real, isolated Chrome session in the cloud, so the browser stays off your laptop.

To run your first session, start free, or

book a demo

for a full walkthrough of the product.