Guide To Web Scraping With JavaScript And Selenium

Benefits of Web Scraping with JavaScript

Below are a few benefits of web scraping:

• Extracting product details of e-commerce websites such as prices, product names, images, etc.
• Web scraping is very useful in research as it can help to gather structured data from multiple websites.
• Gathering data from different sources for analysis can be automated with web scraping easily.
• It can be used to collect data for testing and training machine learning models.

The benefit of web scraping is endless and introduces different thoughtful insights and possibilities to businesses.

Where to check for data protection?

In general, web scraping is not illegal, but there are some caveats to it, and to some extent, it has become illegal.

It’s always very important to be careful and check out some of the different terms you should consider when web scraping to determine if your actions are illegal or not.

• Copyright InfringementEvery website should have a document stating whether the website’s content can be web scraped or not. You can check the copyright document to gain more insights into the possibilities provided by the company on web scraping.
• Robot.txtThe robot.txt file also provides strict information on web scraping, and you should adhere to this file’s information or legal stand. If it says no web scraping, then it should be respected.
• Using APIIf a business provides a public API that users can access the data in the business, then you should adhere to using the API instead of web scraping it to avoid legal action.
• Terms of Services

  Terms of service is a good place to find all the information on how the website’s content should be used and accessed. It states the possibilities of web scraping the stand the website is taking on.

• Scraping Public ContentWeb scraping of public content is not a crime, but if the content owner forbids it – web scraping companies should comply.

Now that we understand our stand on the legality of web scraping, what methods are there to web scrape data either for a small use case or web scraping at scale? Next, let’s explore the different methods and tools available for web scraping.

Implementing Web Scraping with JavaScript and Selenium

One of the benefits of Selenium is that it can web-scrape dynamic JavaScript pages where there are dynamic interactions such as hovering over menu items. In this section of this blog on web scraping with JavaScript and Selenium, we will demonstrate how to web scrape YouTube videos and display the data as JSON using Selenium and JavaScript.

Selenium encapsulates the complexity of building an automation tool for dynamic pages and makes it easy by simplifying installing and configuring the package.

Prerequisites

Before you continue below, you should be familiar with Node.js and Express.js – a Node.js framework. If you are new to Express testing, you can learn more about it through this guide on getting started quickly with Express testing.

Installation

We will continue by installing the necessary packages and software to implement web scraping with JavaScript and Selenium.

• Visual Studio Code: This is the code editor we will use in this demo, though you can go with any code editor of your choice. You could use any other IDE of your choice.
• Browser Driver: To run the Selenium testing drivers on your machine, you need to download the drivers from the following locations according to the browser of your choice.

After downloading any of the drivers, make sure to install it and set it up properly according to your operating system.

After successfully installing and configuring the browser driver, we will explore how to create a new project and start scraping YouTube videos in the next section of this blog on web scraping with JavaScript and Selenium.

Creating a Project

After the installation, we will create a new VSCode Node.js project and add the following libraries to our package.json file.

• Ensure you have Node.js installed and configured properly in your operating system. You can download and install it from the Node.js official website.
• Let’s install the following packages:

```bash
    npm install express selenium-webdriver
    ```


• After installing the libraries, let’s create an index.js file inside the root directory and set up our express server.

```bash
    touch index.js
    ```

Create the express server to listen to incoming requests:

```js
    const express = require('express')
    const { Builder, By } = require('selenium-webdriver');
    
    const app = express()
    const port = 3000
    app.get('/', (request, response) => {
     // Web Scraping Code here
    })
    app.listen(port, () => {
      console.log(`Example app listening at http://localhost:${port}`)
    })

• The code snippet above creates a new Express server and listens to incoming requests from the user. In the next section of this blog on web scraping with JavaScript and Selenium, we will use the selenium-webdriver library we installed to web scrape our YouTube videos.

YouTube Web Scraping with JavaScript and Selenium

In this demonstration, we scrape the following data from the TestMu AI YouTube channel and display it as JSON:

• Video Title
• Video Views
• Video Published Date

You can implement it using the cloud-based Selenium automation grid provided by TestMu AI, which will take away all the manual configuration both in development and in production.

TestMu AI is a cross browser testing platform that allows you to automate browsers with Selenium, Appium, and other frameworks. With TestMu AI’s Selenium Grid, you can run Selenium tests in parallel over an online browser farm of 3000+ browsers and platform configurations.

Below is a simple walkthrough of how the scraping is going to happen before delving into the implementation.

• Open the YouTube channel videos page.
• It scrolls until the end of the page to scrape all the videos available.
• Retrieves the video title, views, and upload details using a loop.
• Convert the scraped information to JSON.

Implementation:

const express = require('express');
const { Builder, By } = require('selenium-webdriver');
 
const app = express();
const port = 3000;
app.get('/', async (request, response) => {
 // Web Scraping Code here
 try {
   const data = await WebScrapingLocalTest();
   response.status(200).json(data);
 } catch (error) {
   response.status(500).json({
     message: 'Server error occurred',
   });
 }
});
app.listen(port, () => {
 console.log(`Example app listening at http://localhost:${port}`);
});
 
async function WebScrapingLocalTest() {
 try {
   driver = await new Builder().forBrowser('chrome').build();
   await driver.get('https://www.youtube.com/c/LambdaTest/videos');
   const allVideos = await driver.findElements(
     By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
   );
   return await getVideos(allVideos);
 } catch (error) {
   throw new Error(error);
 } finally {
   await driver.quit();
 }
}

Code Walkthrough:

Let’s walk through the code together and understand the nitty-gritty of it.

Step 1: Add required packages

First, we require the express and selenium-webdriver packages, and next we created an Express server to handle the incoming request. The request is handled by calling the WebScrapingTest function.

```js

app.get('/', async (request, response) => {
 // Web Scraping Code here
 try {
   const data = await WebScrapingLocalTest();
   response.status(200).json(data);
 } catch (error) {
   response.status(500).json({
     message: 'Server error occurred',
   });
 }
});
 
async function getVideos(videos) {
 let videoDetails = [];
 try {
   for (const video of videos) {
     const title = await video.findElement(By.id('video-title')).getText();
     const views = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[1]"))
       .getText();
     const date = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[2]"))
       .getText();
     videoDetails.push({
       title: title ?? '',
       views: views ?? '',
       publishedDate: date ?? '',
     });
   }
 } catch (error) {
   console.log(error);
 }
 return videoDetails;
}

Step 2: Create an instance of Selenium WebDriver

Inside the openChromeTest function, we created an instance of the Builder class from the Selenium Webdriver package we imported earlier. We also pipe the forBrowser method used to indicate the browser to use for the web scraping and lastly, we call the build method, which creates a new WebDriver client based on this builder’s current configuration.

With the instance of the builder, we passed in the official TestMu AI YouTube videos URL in the get method of the builder class.

```js
    
    driver = await new Builder().forBrowser('chrome').build();
       await driver.get('https://www.youtube.com/c/LambdaTest/videos');

Step 3: Load all the YouTube Videos on the pageNext, we will load all the videos within the page and collect specific information using a loop in JavaScript.

To do that, we need to target a specific element in the YouTube Videos page that contains all the videos using the Chrome inspector tool, we can select any element within the webpage. Below is a screenshot showing the element and the CSS selector to use.

As of the time of writing, the class attribute to target is ytd-grid-video-renderer style-scope ytd-grid-renderer, which is a combination of the CSS class property of the parent container and the individual video container.

Next, we are going to convert the classes to CSS QuerySelector and target all the videos using the FindElements method in the Selenium WebDriver and By instance class. Here is how it looks:

const allVideos = await driver.findElements(
         By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
       );

Selenium uses the By instance class to represent different query selectors, so you can select elements using Selenium locators like CSS name, ID, Class, etc.

Finally, the findElements method returns an object of all the selected HTML DOM Nodes called WebElement, which we will loop through to retrieve metadata of the videos while performing web scraping with JavaScript and Selenium.

Step 4: Converting to JSON

Lastly, we loop through the WebElement and retrieve metadata of the videos such as the video title, video views, and video published date.

 async function getVideos(videos) {
 let videoDetails = [];
 try {
   for (const video of videos) {
     const title = await video.findElement(By.id('video-title')).getText();
     const views = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[1]"))
       .getText();
     const date = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[2]"))
       .getText();
     videoDetails.push({
       title: title ?? '',
       views: views ?? '',
       publishedDate: date ?? '',
     });
   }
 } catch (error) {
   console.log(error);
 }
 return videoDetails;
}

In the code above, we are using both the XPath selectors to accomplish our task of retrieving basic video information. This screenshot below shows you how to retrieve the XPath of a particular element in a webpage.

Secondly, We are also using the ID selector to retrieve the video title. This screenshot shows where to get the ID for the video title element on YouTube using the Inspector tool:

Display the JSON data

After successfully performing YouTube web scraping with JavaScript and Selenium, we convert them into JSON and send them as a response to our request using the Express server. Finally, we are ready to test the new web scrapper.

• Start a new development server using the following command:

```bash
    node index.js
    ```

• Open the Chrome browser and visit localhost:3002, assuming you use the same port number from the demo.
• You should be greeted with JSON data of the YouTube videos after a few seconds, like the screenshot below:

A screenshot showing the result of web scraping with JavaScript and Selenium on the TestMu AI YouTube videos page.

If you love testing as much as I do, you’re going to enjoy learning more about one of the most powerful tools I know of – Selenium WebDriver with JavaScript. This video tutorial covers everything you need to know.