How To Create Automated Web Bot With Selenium In Python

Tools For Creating Web Bot using Selenium with Python

One of the primary objectives of creating a Selenium with Python bot is to leverage the capabilities offered by Selenium for automating interactions with WebElements in the DOM. There is a suite of Selenium testing tools for automating web browsers, which can be used for a range of automation tasks like web scraping, web automation testing, cross browser testing, and much more.

So, let’s look at the setup essentials that are required for creating a web bot using Selenium with Python.

Selenium

The Python library that wraps the Selenium WebDriver provides binding and a simple API to write functional tests for automating web tasks such as clicking buttons, navigating web pages, and filling out forms.

The test script does not interact directly with the web browser. Selenium WebDriver is the core of the Selenium framework. As mentioned in this Selenium WebDriver tutorial, Selenium WebDriver is instrumental in establishing the connection with the browser using its corresponding browser driver. However, you can further deep dive into what is Selenium, if you want to know more about Selenium and its components.

If the tests have to be run on a local Selenium grid, the corresponding browser driver has to be installed on the target machine:

Downloading and installing the browser drivers are required only when the tests have to be performed on the local Selenium Grid. In the case of a Remote Selenium WebDriver, the tests are executed on a cloud Selenium Grid (e.g. TestMu AI).

Read – Benefits Of Cloud Selenium Testing

Python Automation Frameworks

Python for Selenium Automation Testing is a preferable choice since the language is less verbose, supports dynamic typing, supports hassle-free reporting, and much more. The wide range of Selenium with Python frameworks support parallel testing in Selenium; thereby expediting the time to market. Also, creating a Web Bot using Selenium with python is very Simple and easy to implement process.

In this Selenium with Python tutorial, I have touched upon the essentials of Selenium with Python, let’s me, deep dive, into the most popular Selenium with Python frameworks for automation testing:

PyUnit (Unittest)

Unittest (or PyUnit) is the default Python testing framework that comes with the Python package. This testing framework for Python is inspired by the popular JUnit framework. Creating a Web Bot using Selenium with Python Can be achieved with Unittest framework.

Unittest provides simple and flexible test case execution with the quick generation of test reports. Unittest uses camelCase naming conventions instead of Python’s snake_case naming conventions.

Selenium PyUnit tutorial can be a good starting point for someone looking to leverage the framework for web automation testing.

Read – Detailed JUnit Selenium Tutorial

PyTest

PyTest is an open-source testing framework that supports unit testing, functional testing, and API testing. It is available on Python version 3.5 (and higher). Making a Web Bot using Selenium with Python Can be accomplished with PyTest.

Page Object Model (POM) in Selenium Python is instrumental in improving the maintainability and reusability aspects of the source code. I prefer the PyTest framework over PyUnit due to the rich set of plugins that are a boon to the automation engineers.

Some of the most popular PyTest plugins for automation engineers are below:

• pytest-randomly
• pytest-cov
• pytest-django
• pytest-bdd

Selenium PyTest tutorial deep dives into all the integral aspects of web automation testing with Selenium and PyTest.

This PyTest Tutorial for beginners and professionals will help you learn how to use the PyTest framework with Selenium and Python for performing Selenium automation testing.

Robot Framework

The Robot Framework is an open-source automation framework that allows the creation of human-readable test cases using Gherkin.

Watch this video to learn the basics of the robot framework for automation testing. This video will give you an understanding of the various functionalities of the Robot Framework with live examples. Develoers and tester mainly use Robot Framework to make a Web Bot using Selenium with Python as it as various functionalities.

The Robot framework is best suited to leverage the benefits of Behaviour Driven Development (BDD) using Selenium and Gherkin. Robot is also used for acceptance of Test-Driven Development (TDD) and Robotic Process Automation (RPA).

Read – Robot Framework Tutorial For Web Automation Testing

The core implementation of Robot Framework is in Python but can also run on Jython and IronPython. It natively doesn’t support parallel testing but provides a variety of extensions and libraries which are easy to use.

Watch this video to learn about the built-in tools and standard libraries.

For demonstrating the implementation of Selenium Python bot, I am going to use the PyTest framework.

Behave

Behave is testing frameworks used for Behavior-Driven Development (BDD). It has a large amount of documentation and support from the community. To top it all, it provides inbuilt integrations of Django and Flask – two of the most popular web frameworks in Python. After Robot framework Behave to make a Web Bot using Selenium with Python.

Behave is also useful for black box testing. BDD with Behave and Selenium comes with its own set of advantages & shortcomings but it is still a nifty Python automation testing framework for Selenium testing.

Apart from these frameworks that I normally use for web automation testing, there are a number of popular Selenium Python frameworks that can be looked into to leverage the capabilities of Selenium and Python.

Task Schedulers in Python

Task Schedulers are programs that enable scheduling of tasks or scripts after certain time intervals. Alternately, scripts can also be scheduled (or triggered) based on certain conditions.

Task Scheduler can be used extensively for running a batch (or .bat) file that triggers the corresponding Python script.

Some of the common programs that provide task scheduling in Python are:

Cron

Cron is a time-based job scheduler in Unix-like computer operating systems such as Linux and macOS. It allows users to schedule tasks such as taking backups, create logs, and run scripts

Windows Task Scheduler

Task Scheduler is a component of Windows OS that lets you create and run essentially any task in an automatic fashion. Usually, the system and certain apps are scheduled for automation and maintenance. Tasks like disk defragmentation, disk cleanup, and updates, etc. can be scheduled automatically using the Windows Task Scheduler.

Windows Task Scheduler works by keeping track of the time and events on the computers and executes the task if the necessary conditions are met.

For this blog, I am going to use the Windows Task Scheduler program, along with the Win32com Library – a thin Python wrapper that helps automate Windows applications. In this case, it’s the Task Scheduler.

Now that we have touched upon the integral aspects of building a web bot using Selenium with Python, let’s get our hands dirty with some implementation.

2M+ Devs and QAs rely on TestMu AI

Deliver immersive digital experiences with Next-Generation Mobile Apps and Cross Browser Testing Cloud

Sign up with googleSign up with Email

This certification is for professionals looking to develop advanced, hands-on expertise in Selenium automation testing with Python and take their career to the next level.

Here’s a short glimpse of the Selenium Python 101 certification from TestMu AI:

Building Automated Web Bot using Selenium with Python

Follow the below mentioned steps to create a development environment for the implementation:

Creating a Virtual environment and installing dependencies

Virtual environment (or virtualenv) is a tool for creating isolated Python environments. For this purpose, Python provides a venv module that supports creation of lightweight “virtual environments” with its isolated site directories.

You can check out this Selenium Python tutorial for a quick refresher on web automation testing with Selenium and Python.

python3 -m venv env

1. Run the below mentioned command to activate this virtual environment:

Windows

.envScriptsactivate

Linux

source env/bin/activate

2. Run the following command to install Selenium for Python:

pip install selenium

3. Run the following command to install the PyTest framework:

pip install pytest

4. Install the pywin32 package to for managing the Windows Task Scheduler:

pip install pywin32

Folder structure of Test Scripts

The project majorly consists of two files:

• main.py – Contains the main script that enables logging on to the TestMu AI platform and fetching details of the respective tests.
• scheduler.py – Contains the implementation to schedule the test scripts.

Implementation of Automated Selenium Python bot

For demonstrating the creation of Web Bot using Selenium with Python, we automate the test scenario of logging on to the TestMu AI platform. The URL under test is https://accounts.lambdatest.com/login

Since the test is demonstrated on the TestMu AI Grid, I am using the Remote WebDriver in Selenium instead of the local WebDriver. To get started, we import the necessary packages.

from selenium import webdriver

We define two variables that point to the username and access key, the combination of which is used for accessing the TestMu AI cloud grid. The username and access key can be accessed from TestMu AI profile page.

username = "<YOUR USERNAME>"
access_key = "<YOUR ACCESS KEY>"

Define a class named ‘TestClass’ with the following functions:

• Setup_method – It is a function used for setting up the remote driver and its corresponding properties.

# Generate capabilities from here: https://www.lambdatest.com/capabilities-generator/
# setUp runs before each test case and
def setup_method(self):
desired_caps = {
       "build": "your build name",
       "name": "your test name",
       "platform": "Windows 10",
       "browserName": "Chrome",
       "version": "92.0",
       "selenium_version": "3.13.0",
       "geoLocation": "IN",
       "chrome.driver": "91.0",
}
"""
Setup remote driver
       -------
       username and access_key can be found on the lambdatest platform
 
"""
self.driver = webdriver.Remote(
       command_executor="https://{}:{}@hub.lambdatest.com/wd/hub".format(
       username, access_key
       ),
       desired_capabilities=desired_caps )

• Teardown_method – It is a function used for tearing down runs after each test case.

def teardown_method(self):
       self.driver.quit()

• Test_login – Implementation under this function is used for logging into the TestMu AI dashboard. Here, I make use of SendKeys in Selenium to fill out the login form so that I can login to the TestMu AI website. Here is the syntax of the SendKeys method in Selenium Python:

element.send_keys("text to enter")

For locating the respective WebElements, I have used the XPath locator in Selenium. However, you can use other popular Selenium locators like ID, Name, linktext, etc. for locating the WebElements.

Read – XPath Locators Cheat Sheet

Shown below is the syntax of find_element_by_xpath() method in Selenium Python. Alternatively, you can also use the driver.find_element(By.XPATH) method for locating the element using XPath locator.

element = driver.find_element_by_xpath("xpath_here")
 
driver.find_element(By.XPATH, "xpath")

Here is how the entire login method (i.e. test_login) looks like:

def test_login(self):
       """
       this function logins you in lambdatest dashboard
       """
       LOGIN_URL = "https://accounts.lambdatest.com/login"
       DASHBOARD_URL = "https://accounts.lambdatest.com/dashboard"
       self.driver.get(LOGIN_URL)
       login_email = self.driver.find_element_by_xpath(
           "//input[@id='email']"
       )
       login_password = self.driver.find_element_by_xpath(
           "//input[@id='password']"
       )
       login_button = self.driver.find_element_by_xpath(
           "//button[@id='login-button']"
       )
 
       login_email.send_keys(email)
       login_password.send_keys(password)
       login_button.click()
 
       # if we are on right page
       if self.driver.current_url == DASHBOARD_URL :
           assert True
           # time to get recent tests
           self.get_recent_tests()
       else:
           print(" something went wrong !!")
           assert False

• Get_recent_tests – Once I have logged into the TestMu AI Dashboard, I fetch all recent tests from the lambdatest dashboard.

As seen above, .cut-text is a common class for all tags that we are interested to read. Next, I will perform web scraping using Selenium and Python for fetching the required information.

titles = self.driver.find_elements_by_class_name("cut-text")

Here, I clean and scrape text from the respective tags:

def get_recent_tests(self):
       titles = self.driver.find_elements_by_class_name("cut-text")
       titles_list = []
       for i in titles:
           titles_list.append(i.text)
       message = ' '.join(titles_list[0::2])
       print(message)
       send_email(message)

As seen in the implementation, we are using Selenium 3 Grid for testing. With the release of stable Selenium 4, many QA engineers are migrating Selenium automation tests to Selenium 4 grid. Before doing so, I recommend reading Selenium 3 vs Selenium 4 comparison so that tests can be migrated in a seamless manner.