How To Get HTML Source Of A Web Element In Selenium Using Python?

What is an HTML Page Source?

An HTML (Hypertext Markup Language) Page Source refers to the complete set of HTML code that constitutes a web page.

HTML source code of a webpage can be accessed by right-clicking on the page within most web browsers and choosing the option “View Page Source”. This allows users to inspect the underlying code that defines the structure and content of the webpage. Most importantly, testing HTML code in a cross browser testing environment becomes vital for testers and developers.

It is worth noting that the HTML <source> tag is not the same as the HTML page source.

What is an HTML Web Element?

An HTML Web Element is any HTML tag that constitutes the HTML page source code. It can be an HTML code block, an independent HTML tag like </br>, a media object on the web page – image, audio, video, a JS function, or even a JSON object wrapped within <script> </script> tags. These HTML web elements may have unique identifiers like IDs, names, or unique classes.

These elements can be extracted for testing using mechanisms like XPath in Selenium or CSS Selectors.

How to Get HTML Source of WebElement in Selenium WebDriver Using Python?

The get_attribute method in Selenium Python WebDriver is used to get the HTML source of the Web Element. Let’s understand by example in which we will grab and print the source code of the div containing the categories of products with id “entry_213259” as shown below in the given image.

Code:

test.py

from selenium import webdriver

driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://ecommerce-playground.lambdatest.io")
element_Source = driver.find_element("id","entry_213259").get_attribute("outerHTML")
print(element_Source)
driver.quit()

Output Screen:

Code Walkthrough:

We will use Selenium’s find_element method to locate an HTML element by its ID. This line extracts the outer HTML content of the found element (in our case, the element with ID “entry_213250”) and assigns it to the variable element_Source.

Note: We can use various locators, such as id, name, XPath, and CSS Selector, to find the element depending on requirements.

How to Retrieve JSON Data From an HTML Page in Selenium WebDriver Using Python?

Modern applications often rely on multiple APIs, with dynamic content updates using JSON objects instead of XML responses. As a proficient Selenium Python tester, it’s essential to handle JSON objects, particularly those embedded within `<script>` HTML tags. Python offers a built-in JSON library for effective experimentation with JSON objects. Let’s see an example of fetching the JSON data from an HTML Page Source.

Code:

test.py

from selenium import webdriver
import json

# Initialize Selenium WebDriver (you need to specify the path to your webdriver executable)
driver = webdriver.Chrome
driver.maximize_window()
# Load the website
driver.get("https://ecommerce-playground.lambdatest.io")

# Extract the HTML source
html_source = driver.page_source

# Locate JSON data within the HTML source
start_index = html_source.find("window.PRODUCTS = ")
end_index = html_source.find("};", start_index) + 1

# Extract the JSON data
json_str = html_source[start_index + len("window.PRODUCTS = "):end_index]

# Parse the JSON data
json_data = json.loads(json_str)

# Now you can work with the JSON data
print(json_data)

# Close the WebDriver
driver.quit()

Output Screen:

Code Walkthrough:

We will import all the libraries to get the JSON data from an HTML page source.

We will create a new instance of Chrome driver, launch the driver in full-screen mode, and load the TestMu AI E-commerce Website home page.

We will find the JSON objects in the HTML page source of the TestMu AI E-commerce Website.

Now, we will load and print the JSON data from the HTML page source.

Lastly, we will quit the driver.

Retrieving The Earlier JSON Data From an HTML Page Source Using Selenium Cloud Grid

Cloud Selenium services automatically update their browser environments. Extracting HTML page source from up-to-date browsers ensures compatibility and helps identify issues related to the latest browser versions. We will maintain a hybrid testing approach, enabling seamless transitions between local and cloud-based environments, such as TestMu AI cloud platform.

TestMu AI is an AI-powered test orchestration and execution platform. TestMu AI’s scalable infrastructure make it a reliable and cost-effective solution for obtaining HTML source code with the latest compatibility and efficiency.

Follow these easy steps before running a Python test on TestMu AI:

• Create an account on TestMu AI.
• Navigate to your TestMu AI build directory.
• Select “Access Key” located at the top-right corner.
• Copy both your Username and Access Key from the displayed popup modal.

Code:

test.py

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
import json

username = "username"
accessToken = "accessToken"

lt_options = {}
lt_options["video"] = True
lt_options["build"] = "Input Field Test"
lt_options["project"] = "Field Test"
lt_options["name"] = "Test1"
lt_options["w3c"] = True

gridUrl = "hub.lambdatest.com/wd/hub"

url = "https://"+username+":"+accessToken+"@"+gridUrl

options = ChromeOptions()
options.set_capability('LT:Options', lt_options)

driver = webdriver.Remote(  command_executor= url, options=options)
driver.maximize_window()
driver.get("https://ecommerce-playground.lambdatest.io/")
html_source = driver.page_source

# Locate JSON data within the HTML source
start_index = html_source.find("window.PRODUCTS = ")
end_index = html_source.find("};", start_index) + 1

# Extract the JSON data
json_str = html_source[start_index + len("window.PRODUCTS = "):end_index]

# Parse the JSON data
json_data = json.loads(json_str)

# Now you can work with the JSON data
print(json_data)

# Close the WebDriver
driver.quit()

Output Screen:

Code Walkthrough:

Add your credentials in this section to enable TestMu AI to execute tests using your account. Get the username and access key from the TestMu AI Dashboard, then copy and paste them into your code.

Access your desired capabilities that can be generated from the TestMu AI capabilities generator.

Provide the remote URL for running a test over the TestMu AI Selenium Cloud.

Conclusion

In conclusion, retrieving HTML page sources in Selenium using Python is essential for identifying and rectifying issues across various website testing scenarios. The ability to dissect and analyze the HTML page source provides testers with a powerful tool to enhance the reliability and resilience of web applications. This blog covered all the essential ways to extract the HTML page source and web element in Selenium using Python to enhance the quality and resilience of modern web systems.