Dive Into Headless Chrome: Your Key to Efficient Web Development

Different Versions of Headless Chrome

A new mode was added in Chrome version 96 along with the established –– headless option. From versions 96 to 108, this fresh mode was known as –– headless=chrome, and from versions 109 onwards, it was known as –– headless=new. Users have complete access to all browser features in this mode, including the ability to use extensions. This improvement goes above and beyond what the standard headless mode can do, giving users and developers additional freedom to use Chrome’s robust features while operating in a headless environment.

Working of Chrome –– headless=new Mode

Before we delve into understanding – – headless=new mode, it’s essential to grasp the working of –– headless. From the implementation of Headless Chrome, it can easily be misinterpreted that the Headless Chrome and regular Chrome browser are the same. Interestingly, this was only partially true.

The codebase of Chrome Headless didn’t share any of the Chrome browser code located in //chrome. However, Headless mode introduced some unique bugs and features that are not found in regular Chrome. This sometimes creates a scenario where a test case is working correctly in one mode but fails in the other. It also slowed automation testing performance involving browser extensions or other browser-level functions without dedicated Headless support.

In 2021, the Chrome development team finally integrated the Headless and Headful modes in version 112. In this mode, Chrome starts but doesn’t display any platform windows. However, all existing functionalities and future updates are available with no restrictions.

Basic Headless Chrome Techniques

This section covers some basic functionalities like detecting, getting the title of the website, taking screenshots, and many more.

How to Detect Headless Chrome in Selenium

You can use userAgent to detect whether Chrome is running in Headless mode or not. A web browser or other client software transmits a userAgent string to a web server with each request to identify itself and its capabilities. The user agent string includes details that can be used to determine the client, such as the operating system, device type, and browser version.

The below code shows the use of userAgent for detection.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

options = ChromeOptions()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

isHeadless = driver.execute_script("return navigator.userAgent;")
substr = "HeadlessChrome"

print(isHeadless)
if substr.lower() in isHeadless.lower():
    print("Headless Chrome detected")
else:
    print("Non-Headless Chrome detected")

driver.quit()

Console Output:

Getting the Title of the Website

The title of the website can be used to verify that the website is accessible to users and also verify that the website is loading the correct page.

The below code shows how to get the title and check it using assertion.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

# Create a new ChromeOptions object.
options = ChromeOptions()

# Set the headless mode.
options.add_argument("--headless=new")

# Create a new ChromeDriver object, passing in the ChromeOptions object.
driver = webdriver.Chrome(options=options)

# Navigate to the desired website.
driver.get("https://ecommerce-playground.lambdatest.io/")

# Print the title of the page.
print("Title of the page is: " + driver.title)

# Assert that the title of the page contains the text "Your Store".
assert "Your Store" in driver.title

# Close the browser.
driver.quit()

Console Output:

The title method is used to get the title of the website. Assert is used for verifying or validating the website title. Learn How To Use Assert and Verify in Selenium WebDriver for accurate validation of website titles.

Taking a Screenshot of a Window

The screenshot ensures that while performing automation tests, we can have a visual record of the page’s state in case of any errors.

The below code shows how to use the save_screenshot() method.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

options = ChromeOptions()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

driver.get("https://ecommerce-playground.lambdatest.io/")

# Save a screenshot of the current page to the file "screenshot.png".
driver.save_screenshot("screenshot.png")
driver.quit()

Taken Screenshot:

Selenium by default provides a method to record screenshots via the save_screenshot() method.

Advanced Headless Chrome Techniques

For developers, testers, and QAs, advanced techniques in Headless Chrome open up a world of customization possibilities. We’ll look at some of the methods in this section, like customizing userAgent, handling cookies, printing Document Object Model (DOM), generating PDFs, and working with ActionChains.

Customizing userAgent

Customizing the userAgent serves multiple purposes, such as debugging web applications, testing web applications across various browser environments, enabling web scraping without triggering anti-scraping measures, and accessing websites that may be restricted based on geographical location or device types.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

# Define your custom user agent
custom_user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/116.0.0.0 Safari/537.36"
 
options = ChromeOptions()
options.add_argument("--headless=new")
options.add_argument(f"--user-agent={custom_user_agent}")  # Set custom user agent

driver = webdriver.Chrome(options=options)

user_agent = driver.execute_script("return navigator.userAgent;")
# Print the user agent string.
print(user_agent)
driver.quit()

Console Output:

Handling Authentication and Cookies

While performing web testing, it becomes crucial to handle and use cookies related to authentication or maintaining session data. Selenium provides various functions like add_cookie(), get_cookie(), get_cookies() and delete_cookie() to work with cookies.

Learn How To Handle Cookies in Selenium WebDriver in detail.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

driver.get("https://ecommerce-playground.lambdatest.io/")

driver.add_cookie({"name": "rohan", "value": "user1234"})
print(driver.get_cookie("rohan"))
driver.quit()

Console Output:

Capturing Screenshots

Taking snapshots of different HTML elements like div, span, a, and many more while performing tests helps to figure out issues related to UI and rendering data.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

driver.get("https://ecommerce-playground.lambdatest.io/")

element = driver.find_element(By.XPATH,'//*[@id="mz-carousel-213240"]/div/div[1]/a/img')

element.screenshot('screenshot.png')

driver.quit()

Taken Screenshot:

Printing the DOM

Printing Document Object Model (DOM) helps evaluate a web page’s structure, which is essential for debugging and guaranteeing data consistency. You can inspect the page’s structure and spot any irregularities in the data retrieving process by printing the DOM.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.common.by import By
import time
options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

driver.get("https://ecommerce-playground.lambdatest.io/")

print(driver.page_source)

# or

html = driver.execute_script("return document.documentElement.outerHTML;")

# Save the DOM to a file.
with open("dom.html", "w") as f:
    f.write(html)
driver.quit()

Console Output:

File Output:

Generating PDFs

Generating PDFs is a handy method for archiving web content and verifying that web pages are printable. There are various ways to generate a PDF:

Using the Selenium print_page() method: The print_page() prints the current page within the browser. Read more about it from the official docs of selenium.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.common.print_page_options import PrintOptions
import base64
print_options = PrintOptions()
print_options.page_ranges = ['1-2']

options = ChromeOptions()
options.add_argument('--headless=new')

driver = webdriver.Chrome(options=options)
driver.get("https://ecommerce-playground.lambdatest.io/")

base64code = driver.print_page(print_options)
# decode the base64 string to bytes
pdf_bytes = base64.b64decode(base64code)

# write the bytes to a file with a .pdf extension
with open('output.pdf', 'wb') as f:
    f.write(pdf_bytes)
driver.quit()

Console Output:

Using Chrome DevTools Protocols:

Chrome DevTools Protocols (CDP commands) allow you to interact with and control Google Chrome or Chromium-based browsers programmatically. You can use CDP commands to perform various tasks, such as navigating to web pages, inspecting the DOM, capturing screenshots, and more. Here’s an overview of how to use CDP commands with the help of the execute_cdp_cmd() method provided by Selenium:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
import base64


options = ChromeOptions()
options.add_argument('--headless=new')

driver = webdriver.Chrome(options=options)
driver.get("https://ecommerce-playground.lambdatest.io/")

# Execute the CDP command to print the page to PDF.
pdf_data = driver.execute_cdp_cmd("Page.printToPDF", {"path": 'html-page.pdf', "format": 'A4'})

# Decode the PDF data and write it to a file.
with open('cdp_output.pdf', 'wb') as f:
    f.write(base64.b64decode(pdf_data['data']))

driver.quit()

Taken Screenshot:

Capture, Create, and Recreate Actions

Automating user interactions allows you to create end-to-end test scripts, replicating real user behavior.

• Capture: Use ActionChains to record user actions like mouse movements and clicks; call the perform() method to execute them.
• Create: Make a new ActionChains object, add user actions, and call the perform() method to execute them.
• Recreate: To repeat captured user actions, create new ActionChains, add recorded actions, and call perform().

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains

options = ChromeOptions()
options.add_argument('--headless=new')

driver = webdriver.Chrome(options=options)

driver.get("https://ecommerce-playground.lambdatest.io/")

# Set the window size to 1200x760
driver.set_window_size(1200, 760)

# Find the element to interact with
element = driver.find_element(By.CSS_SELECTOR, ".active > .d-block > .d-block")

# Create a new ActionChains object
actions = ActionChains(driver)

# Move the mouse to the element and click and hold
actions.move_to_element(element).click_and_hold().perform()

driver.save_screenshot('action1.png')

# Release the mouse click
actions.move_to_element(element).release().perform()

# Click on the dropdown toggle
driver.find_element(By.CSS_SELECTOR, ".dropdown-toggle > .info:nth-child(2) > .title").click()

driver.save_screenshot('action2.png')

driver.quit()

Action1 Image Output:

Action 2 Image Output:

Retrieving Performance Logs from Chrome Headless

It can be retrieved using execute_cdp_cmd() in Selenium. Performance logs are essential because they provide information about how a web application is being used, and help locate and fix performance issues, resulting in faster loading of web applications and ultimately better user experiences.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

options = ChromeOptions()
options.add_argument('--headless=new')

driver = webdriver.Chrome(options=options)
driver.get("https://ecommerce-playground.lambdatest.io/")

driver.execute_cdp_cmd('Performance.enable', {})
t = driver.execute_cdp_cmd('Performance.getMetrics', {})
print(t)

driver.quit()

Console Output: