How to Scrape Amazon Product Data With Playwright

Why an Amazon Scraper Breaks (and What Fixes It)

Before writing code, it helps to know the three failure modes that kill most scrapers, because every technical choice in this tutorial is a direct answer to one of them.

• The data is not in the raw HTML. Product pages hydrate with JavaScript, so a plain request sees an empty shell. The fix is a real browser that executes JavaScript before you read the DOM.
• Selectors rot. A locator tied to a deep CSS chain or an auto-generated class name breaks the moment the markup shifts. The fix is meaning-based locators that survive layout churn.
• Bot defenses trigger. Aggressive detection blocks or challenges automated traffic. The fix is not a magic bypass, it is building for graceful failure and respecting rate limits.

Playwright answers the first two directly. It drives a real Chromium, Firefox, or WebKit browser, and its Playwright locators are, in the words of the official docs, the central piece of Playwright's auto-waiting and retry-ability. That auto-waiting is why a well-built locator reads data that is not on the page yet without a fixed sleep.

What You Need First

Keep the prerequisites minimal. This tutorial uses Node.js and JavaScript, and every code block is runnable as written.

• Node.js 18 or newer installed, so you can run the script and use the built-in fetch and file APIs.
• A project folder with Playwright installed, plus its bundled browsers.
• Basic JavaScript comfort with async and await, since every browser action returns a promise.

mkdir amazon-scraper && cd amazon-scraper
npm init -y
npm install playwright
npx playwright install chromium

One honest note on the target. Amazon's Conditions of Use restrict automated access, so we demonstrate the technique against the public Ecommerce Playground, which mirrors a real storefront's structure. Every selector strategy here transfers to any product grid; the playground just lets you run the tutorial without tripping a defense you should be respecting anyway.

Launch a Browser and Open a Search

The first script does three things: launches Chromium, opens a search results page, and waits for the product grid to render. Running headed at first (with headless: false) lets you watch the page load and confirm the data actually appears.

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });
  const page = await browser.newPage();

  const query = "apple";
  const url =
    "https://ecommerce-playground.lambdatest.io/index.php?route=product/search&search=" +
    encodeURIComponent(query);

  await page.goto(url, { waitUntil: "domcontentloaded" });

  // Wait for the first product card instead of a fixed sleep
  await page.locator(".product-thumb").first().waitFor();

  const count = await page.locator(".product-thumb").count();
  console.log("Product cards on page:", count);

  await browser.close();
})();

The line that matters is waitFor() on the first product card. Playwright auto-waits for a range of actionability checks, including whether an element is visible and stable, before it acts. Waiting on a real element, not a guessed number of milliseconds, is the single biggest reliability win over a naive scraper.

Find the Right Selectors

A selector is a promise about the page structure. Amazon and any large storefront change markup constantly, so the goal is locators that describe what an element means, not where it currently sits in the tree.

Playwright's docs list the recommended locator methods in priority order, and the pattern generalizes well beyond the playground.

The rule of thumb: locate the repeating card once, then read each field relative to that card. Scoping field lookups inside the card is what stops the price from row two attaching itself to the title from row one.

Extract Structured JSON

Raw text is not the deliverable. You want a clean array of records that a database, a spreadsheet, or a downstream job can consume. Iterate over each product card and build one object per product.

const fs = require("fs");
const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();

  const url =
    "https://ecommerce-playground.lambdatest.io/index.php?route=product/search&search=apple";
  await page.goto(url, { waitUntil: "domcontentloaded" });
  await page.locator(".product-thumb").first().waitFor();

  const cards = page.locator(".product-thumb");
  const total = await cards.count();
  const products = [];

  for (let i = 0; i < total; i++) {
    const card = cards.nth(i);
    const title = (await card.locator(".caption a").first().textContent())?.trim();
    const price = (await card.locator(".price-new, .price").first().textContent())?.trim();
    const link = await card.locator("a").first().getAttribute("href");

    if (title) {
      products.push({ title, price: price || null, url: link || null });
    }
  }

  fs.writeFileSync("products.json", JSON.stringify(products, null, 2));
  console.log("Saved", products.length, "products to products.json");

  await browser.close();
})();

Two details make this robust. Each field is read inside the card with a scoped locator, and the optional chaining plus the if (title) guard skip empty cards instead of pushing junk records. The output is a JSON file you can diff, load, or ship.

Handle Pagination Across Result Pages

One page of results is rarely enough. There are two reliable pagination strategies, and the URL approach is the one to prefer when it exists.

• URL parameter walking: most search pages accept a page number in the query string. Increment it in a loop, which is deterministic and does not depend on a button existing.
• Next-button clicking: when there is no page parameter, locate the next-page control, click it, and wait for the new grid. Stop when the control disappears.

async function scrapeAllPages(page, baseUrl, maxPages = 5) {
  const all = [];

  for (let p = 1; p <= maxPages; p++) {
    await page.goto(baseUrl + "&page=" + p, { waitUntil: "domcontentloaded" });

    // If no cards render, we have walked past the last page
    const hasResults = await page.locator(".product-thumb").first().isVisible().catch(() => false);
    if (!hasResults) break;

    const cards = page.locator(".product-thumb");
    const total = await cards.count();

    for (let i = 0; i < total; i++) {
      const card = cards.nth(i);
      const title = (await card.locator(".caption a").first().textContent())?.trim();
      const price = (await card.locator(".price-new, .price").first().textContent())?.trim();
      if (title) all.push({ page: p, title, price: price || null });
    }

    // Be a polite client: randomized pause between pages
    await page.waitForTimeout(1000 + Math.random() * 1500);
  }

  return all;
}

The maxPages cap and the randomized pause are not optional polish. They keep the scraper from running away into thousands of requests and from hammering the server at a machine-gun cadence, which is exactly the behavior bot defenses are tuned to catch.

The Anti-Bot Reality

Here is the part most tutorials skip. Amazon runs some of the most aggressive bot detection on the public web, and no library, plugin, or cloud feature guarantees a bypass. Anyone selling you certainty is selling you a future outage.

What you can do is behave less like a bot and fail gracefully when you are caught. In practice that means:

• Slow down. Randomized delays between requests and a hard page cap keep your traffic pattern human-shaped.
• Set a real context. A normal viewport, a current user agent, and a realistic locale reduce the number of automated-traffic signals you emit.
• Detect the challenge. Check for a CAPTCHA or block page before you parse, so you log a clear reason instead of writing empty records.
• Rotate network egress where terms of use permit, using a legitimate proxy so you are not funneling everything through one IP. The proxy server explainer covers how that layer works.

const context = await browser.newContext({
  userAgent:
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
    "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
  viewport: { width: 1366, height: 768 },
  locale: "en-US",
});
const page = await context.newPage();

// Detect a block before parsing so failures are explicit, not silent
const blocked = await page.getByText(/enter the characters|robot check/i)
  .first()
  .isVisible()
  .catch(() => false);
if (blocked) {
  console.warn("Bot challenge detected. Backing off.");
}

Cloud browser platforms, including TestMu AI Browser Cloud, offer best-effort stealth features such as fingerprint masking and CAPTCHA handling. The word that matters is best-effort: it improves success rates on resistant sites, it does not make a session undetectable, and it is never a guarantee.

Run It on a Cloud Browser

A single local browser is fine for a demo. Scraping a catalog at any real volume means many browsers running in parallel, which is memory-heavy on a laptop and painful to operate on your own servers. This is the gap TestMu AI Browser Cloud is built to close: it provisions real, full-featured Chrome sessions on demand and records video, console logs, and network logs for every run so a failed scrape is debuggable instead of a black box.

To validate the extraction logic against a real cloud session rather than a local one, the Browser Cloud SDK runs a quick scrape without any session boilerplate. The snippet below is what produced the run captured after it.

const { Browser } = require("@testmuai/browser-cloud");
const client = new Browser();

(async () => {
  const data = await client.scrape({
    url: "https://ecommerce-playground.lambdatest.io/index.php?route=product/search&search=apple",
    format: "html",
    lambdatestOptions: { "LT:Options": { username: "YOUR_USERNAME", accessKey: "YOUR_ACCESS_KEY" } },
  });

  const html = typeof data === "string" ? data : data.html;
  // Parse the fully rendered HTML into structured records here
  console.log("Rendered HTML length:", html.length);
})();

Running that scrape against the Ecommerce Playground on a real cloud Chrome session returned a fully rendered page and 15 product cards in about seven seconds. Parsing the rendered HTML produced this structured output, which is the actual console result of the run:

SCRAPE OK  chars=228575  time=7.4s  products_found=15
[
  { "title": "iPod Shuffle",       "price": "$182.00" },
  { "title": "iPod Nano",          "price": "$122.00" },
  { "title": "Apple Cinema 30\"",   "price": "$122.00" },
  { "title": "MacBook Pro",        "price": "$2,000.00" },
  { "title": "Apple Cinema 30\"",   "price": "$122.00" }
]

That is the whole pipeline in one result: a real browser rendered the page, JavaScript ran, and the parser produced clean records. The Browser Cloud documentation walks through session transparency and the built-in tunnel for reaching staging or private storefronts. Teams that prefer driving the browser from a terminal or a CI job can reach the same infrastructure through the Kane CLI testing agent.

Stay on the Right Side of the Rules

A working scraper is a technical achievement; a defensible one is an operational discipline. Treat these as guardrails, not afterthoughts.

• Read the terms of use first. Amazon's Conditions of Use restrict automated access. The technique in this tutorial is demonstrated on a public playground precisely so you can learn it without violating a real site's terms.
• Prefer official APIs. Where the Amazon Product Advertising API covers your use case, it is more stable and sanctioned than scraping. Check it before you build a browser pipeline.
• Respect robots and rate limits. Honor a site's stated crawl rules and keep request volume modest. Aggressive scraping harms the target and gets you blocked.
• Avoid personal and copyrighted data. Public price and title data is a different risk category than reviews containing personal information. Know which you are collecting.

For scenarios where continuously reading dynamic prices is the whole job, the deep dive on price scraping at scale covers the compliance and throughput tradeoffs in more depth than a single tutorial can.

Next Steps

Start by running the structured-JSON script from the extraction section against the Ecommerce Playground; once it emits a clean products.json, you have the core of a working amazon scraper. Then layer on pagination, add the block-detection check, and only after that consider volume. When you outgrow a single site, the broader guide to ecommerce web scraping covers the production architecture that sits around a scraper like this one.

When one browser is no longer enough, move the same Playwright logic onto real cloud Chrome sessions with TestMu AI Browser Cloud, and use its recorded video and network logs to debug the runs that come back empty. Keep the terms of use in front of you the entire time, prefer an official API where one fits, and treat stealth as best-effort rather than a promise. Build for graceful failure and your scraper will outlast the next markup change.