News Scraping at Scale: Pagination, Paywalls, and Article Extraction

What Makes News Different From Other Scraping?

A product-catalog scrape reads a stable list of items with prices. News reads a moving stream where the same story is republished, the layout is buried under ads, and the most valuable content is often gated. The difficulty is not one hard thing; it is five medium things stacked together.

• Endless feeds: Section fronts increasingly use infinite scroll instead of numbered pages, so there is no ?page=2 to loop over.
• Access boundaries: Paid content sits behind a wall that is an explicit legal line, not a technical puzzle to solve.
• Boilerplate noise: The article body is a small fraction of the DOM, wrapped in navigation, related-links rails, newsletter prompts, and ad slots.
• Syndication clones: One wire report from an agency is republished verbatim across many outlets, so naive collection stores it dozens of times.
• Freshness pressure: A story is worth most in its first hour, so the scraper has to catch new URLs quickly without re-crawling everything.

The distribution shape amplifies the syndication problem. A single wire report from a news agency is licensed to hundreds of outlets and republished, often verbatim, so a broad scrape meets the same story through many mirrored copies rather than one canonical source. That is why the dedup step covered later is not optional once you collect from more than a handful of publishers.

How Do You Handle Pagination and Infinite Scroll?

News feeds come in three pagination shapes, and each has a cleaner path than brute-forcing the DOM. Identify which one you are dealing with before writing extraction code.

• Numbered pages: Increment the page parameter and stop when a page returns no new article links or repeats the previous page. The simplest and most stable case.
• Load-more buttons: A button fetches the next batch. Click it in a real browser, wait for the new items to render, and repeat until it disappears or stops adding rows.
• Infinite scroll: New items load as you scroll. A plain HTTP request only ever sees the first batch, so you need a browser that scrolls and waits for network-driven content.

The reason a browser is unavoidable for the last two is rendering. The median mobile page shipped 558 kilobytes of JavaScript in 2024, and on modern publishers that script is what assembles both the feed items and the article text after the initial load.

One shortcut worth checking first: an infinite-scroll feed almost always talks to a JSON or feed endpoint behind the scenes. Open the network tab, find the request that returns the next batch, and call it directly with the page or cursor parameter it uses. When that endpoint exists, it is faster and less brittle than scrolling the page. When it does not, drive the scroll in a real browser:

// Scroll an infinite feed until it stops growing (Playwright-style)
async function collectFeed(page, { maxRounds = 40, pause = 1200 } = {}) {
  const seen = new Set();
  let stable = 0;

  for (let round = 0; round < maxRounds; round++) {
    const links = await page.$$eval("a[href*='/article/'], a[href*='/news/']",
      as => as.map(a => a.href));
    links.forEach(l => seen.add(l));

    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
    await page.waitForTimeout(pause); // let the next batch render

    const height = await page.evaluate(() => document.body.scrollHeight);
    stable = height === page._lastHeight ? stable + 1 : 0;
    page._lastHeight = height;
    if (stable >= 3) break; // feed stopped growing
  }
  return [...seen];
}

Always set a stop condition. A feed that keeps loading yesterday, last week, and last year will scroll forever, so cap the rounds, stop when the height stabilizes, or stop when you reach articles older than your freshness window. The same rendering challenge shows up in our guide to scraping dynamic web pages.

Where Is the Paywall Line, Legally and Ethically?

A paywall is not a technical obstacle to route around. It is an explicit statement that the content is licensed, not free, and stepping over it can breach the publisher's terms of service and copyright. The honest position is simple: content behind a paywall you have not paid for is off the table.

That still leaves a large, legitimate surface, because most news is free to read. The Reuters Institute Digital News Report 2024 found that the proportion who pay for online news across 20 countries is just 17%, a figure flat for three years. The other side of that number is that the vast majority of published articles are readable without a subscription.

Draw the line clearly before you write extraction code. What follows is engineering guidance, not legal advice; for your specific use case, consult counsel.

• Read robots.txt first: If a publisher disallows a path, treat that as a boundary, not a suggestion, and keep your crawler out of it.
• Respect the terms of use: Many publishers state their scraping and reuse rules explicitly; honor them for each site rather than applying one blanket policy.
• Take the free surface only: Headlines, standfirsts, links, publish dates, and free-to-read body text are the extractable layer; gated full text is not.
• Do not defeat access controls: Clearing a metered counter, spoofing a subscriber session, or bypassing a hard wall crosses the line, even when it is technically easy.
• License full feeds when you need them: If you need complete articles from a paid publisher, buy their content API or syndication feed instead of scraping past the wall.

A well-behaved scraper also rate-limits itself, identifies with an honest user agent, and avoids collecting personal data. Best-effort tooling such as stealth mode reduces friction on sites that block ordinary automation, but it is never a license to cross an access boundary a publisher has set.

How Do You Strip Boilerplate to Get the Article Body?

On a news page, the article you want is a minority of the DOM. Around it sit the masthead, a mega-menu, a related-stories rail, a newsletter prompt, social buttons, a comment widget, and several ad slots. Boilerplate removal is the step that keeps the body and discards the rest.

Two approaches exist, and the second is far more durable across a fleet of publishers.

A readability-style extractor walks the DOM, scores nodes by the ratio of text to links and markup, and keeps the highest-scoring contiguous block as the article. That is why it generalizes: it looks for the shape of an article rather than a named class that a redesign can rename overnight.

The critical detail is when you run it. Run boilerplate removal on the fully rendered page, not the raw HTML shell, because much of both the content and the clutter is injected by JavaScript after load. TestMu AI Browser Cloud can return page content in a readability format directly, which does the density scoring on the rendered DOM for you:

import { Browser } from '@testmuai/browser-cloud';
const client = new Browser();

// 'readability' strips nav, ads, and rails from the rendered DOM,
// returning the main article content instead of the raw shell.
const article = await client.scrape({
  url: 'https://example-news-site.com/world/some-report',
  format: 'readability',
  lambdatestOptions: {
    'LT:Options': { username: 'keys', accessKey: '<your-access-key>' }
  }
});

console.log(article); // headline + clean body, boilerplate removed

For the fields around the body, lean on structured metadata the publisher already exposes. Most news pages embed NewsArticle or Article JSON-LD and Open Graph tags with the headline, author, and publish date, which is cleaner to read than parsing visible text and consistent across sites.

How Do You Dedupe Syndicated Stories?

Wire agencies sell one report to hundreds of outlets, so a broad news scrape collects the same story many times: verbatim in some places, lightly re-topped in others. Without a dedup step, your dataset is mostly duplicates, and any downstream count or model is skewed by the loudest wire copy.

Dedup works in two layers, cheap first, then fuzzy for what slips through.

• Exact match: Normalize the body text (lowercase, collapse whitespace, strip punctuation), hash it, and drop rows whose hash you have already stored. This catches verbatim reposts instantly.
• Near-duplicate match: For edited copies, compute a similarity signal on the title plus lead paragraph, using shingling, MinHash, or a cosine similarity over embeddings, and cluster anything above a threshold.
• Pick a canonical: Keep the earliest-published or the originating outlet's version as the representative, and link the rest as mirrors so you keep provenance without storing redundant bodies.

A cheap first-pass filter you can compute during extraction is a normalized-title hash. It will not catch re-topped headlines, but it removes the exact clones before they ever reach the expensive near-duplicate stage:

import { createHash } from 'crypto';

const seen = new Set();

function fingerprint(title, lead) {
  const norm = (title + ' ' + lead)
    .toLowerCase()
    .replace(/[^a-z0-9 ]+/g, '')
    .replace(/\s+/g, ' ')
    .trim();
  return createHash('sha1').update(norm).digest('hex');
}

function isDuplicate(article) {
  const fp = fingerprint(article.title, article.lead);
  if (seen.has(fp)) return true;   // already have this story
  seen.add(fp);
  return false;
}

In a real run against a rendered listing, this problem is visible immediately: the same items repeat in the extracted rows, exactly the pattern the fingerprint check exists to collapse. You will see that in the extraction run below.

How Do You Schedule for Freshness Without Re-Crawling Everything?

A news story is worth most in its first hour, and worth less every hour after. The scheduling goal is to notice new URLs quickly while spending almost no work re-checking articles you already have.

The pattern is an incremental crawl driven by a seen-set, not a full re-scrape on every run.

• Match cadence to velocity: Poll a breaking-news or markets front every few minutes; poll a weekly column daily. One global interval either wastes requests or misses stories.
• Diff against a seen-set: On each poll, collect the feed's URLs, subtract the ones you already stored, and only fetch and extract the genuinely new ones.
• Prefer feeds and sitemaps: RSS, Atom, and news sitemaps announce new URLs cheaply, so poll them before rendering a full section front.
• Handle updates, not just inserts: Articles get corrected and re-topped, so store a content hash and re-extract when the modified timestamp or hash changes.
• Back off politely: Spread requests, honor rate limits, and slow down on errors so a high-frequency poll never looks like an attack.

Where you want a scheduled check to also validate that a rendered page still looks right, a deterministic browser agent fits the job. TestMu AI Kane CLI is a browser agent for developers, AI agents, and CI/CD that drives a real Chrome session from natural-language objectives, so a scheduled pipeline can confirm a feed rendered and the expected article shell loaded before the extractor runs, then export the steps as reusable code.

What Does a Real Extraction Run Look Like?

To ground the pattern, here is an actual run of the TestMu AI Browser Cloud SDK against a rendered, JavaScript-heavy listing, standing in for a news section front. It renders the page in a real Chrome session and parses listing rows out of the hydrated DOM, the same shape as pulling headlines from a feed.

import { Browser } from '@testmuai/browser-cloud';
const client = new Browser();

const data = await client.scrape({
  url: 'https://ecommerce-playground.lambdatest.io/',
  format: 'text',
  lambdatestOptions: {
    'LT:Options': { username: 'keys', accessKey: '<your-access-key>' }
  }
});

const text = typeof data === 'string' ? data : (data.text || JSON.stringify(data));
const lines = text.split('\n').map(l => l.trim()).filter(Boolean);

// Parse "listing rows" out of the rendered DOM, the way you would
// pull headlines from a news feed.
const rows = lines.filter(l => /^(HTC|iMac|MacBook|iPhone|Samsung|Canon)/i.test(l));
rows.slice(0, 6).forEach((r, i) => console.log(`  ${i + 1}. ${r}`));

The real console output from that run:

[browser-cloud] starting real session against LambdaTest Ecommerce Playground
[browser-cloud] rendered page in 8.4s, extracted 2939 chars of text
[browser-cloud] sample product listings parsed from rendered DOM:
  1. iMac
  2. iMac
  3. iMac
  4. iMac
  5. HTC Touch HD
  6. HTC Touch HD
[browser-cloud] total candidate listing rows found: 6
[browser-cloud] done

Two things in that output map straight to news extraction. First, the content only exists because a real browser executed the page's JavaScript; a raw request would have returned an empty shell. Second, look at the rows: iMac appears four times and HTC Touch HD twice, the exact duplication you get when a syndicated story is mirrored across a feed, which is why the fingerprint dedup step earlier is not optional.

Every session also records video, console, and network logs, so when an extraction returns empty rows you can watch what the page actually rendered instead of guessing. For code examples in your own tests, the Selenium Playground gives you stable, public pages to point a scraper at.

How Do You Ship a News Scraper to Production?

Start with one publisher and one section, and get the full path working before you fan out: discover new URLs from a feed, render each in a real browser, strip boilerplate to the body, fingerprint for dedup, and store with provenance. Only then add more sites, because each new publisher tests your boilerplate and dedup logic, not your plumbing.

Put the rendering on managed infrastructure so scale and debuggability are not your problem. TestMu AI Browser Cloud provisions real Chrome sessions on demand, runs them in parallel, and captures per-session video and logs, and its readability format returns clean article content from the rendered DOM. Wire the SDK following the Browser Cloud documentation, and pair it with a browser-automation workflow like the ones in our browser automation guide.

Keep the guardrails permanent, not an afterthought: read each site's robots.txt and terms, stay off paywalled content you have not licensed, rate-limit and back off on errors, and store the source URL and publish time with every record so provenance survives dedup. A news scraper that is fast, clean, and well-behaved beats one that is merely fast.