Problem with web scraping with selenium in python

I have written code to log in to a website with Python. But it takes very long to implement and does not lead to what is expected and gives a message saying "DevTools listening on". (The send_keys command doesn’t work and will not write “password” and “username” to input boxes).

The code is:

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
username="rahbar"
password="password"
print("something before scrape")
driver = webdriver.Chrome()

driver.get("https://plp.irbroker.com/index.do")
time.sleep(100) #waits 100 seconds
driver.maximize_window()
driver.find_element("name", "j_username").send_keys(username)

driver.find_element_by_name( name="j_password" ).send_keys(password)
print("something after scrape")

The message gotten in the terminal is

something before scrape
*DevTools listening on ws://127.0.0.1:57315/devtools/browser/fd73344f-c857-439e-b20c-c0e76d39f389
Created TensorFlow Lite XNNPACK delegate for CPU.
Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors (tensor#58 is a dynamic-sized tensor)*

I expect the code to open the webpage and write “password” and “username” to input boxes in a reasonable short time period. Can you guide me on this issue (I am new to Python, so sorry if my question is trivial)?

Solution

Let's improve the code before the actual scraping.

It is recommended that all data be moved to the beginning of the code. It is easier to see all the necessary data in one place:

# data for log-in
username = "rahbar"
password = "password"
site_url = "https://plp.irbroker.com/index.do"

Now let's concentrate on optimizing and refining a few points.

First, let's get website login and browser initialization out of the way and add fullscreen mode to the browser startup parameters:

# initialize driver
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(options=chrome_options)
driver.get(site_url)

Now, instead of time.sleep(),let's implement a method that returns the element after it has been displayed on the site, although it is not always accurate and correct:

# wait until both input elements will be on screen
username_element = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='j_username']")))
password_element = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='j_password']")))

If you are entering data into an input element, it is better to use EC.element_to_be_clickable. This method checks not only for the presence of the element on the page, but also for interactivity(i.e.,whether the element is visible or disabled).

Now, let's clear the input field (if necessary) and enter the value:

# clear them (not important is site does not use start value)
username_element.clear()
password_element.clear()
# set values
username_element.send_keys(username)
password_element.send_keys(password)

And finally, we move on to the last part (optional since it wasn't mentioned in the question) now we wait until the login button can be clicked (this works the same as with the input field):

# wait until login button will be clickable and then press it
log_in = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "input[type='submit'][name='B1']"))).click()

Now the code is optimized and clear!

Here is the full version:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options

# data for log-in
username = "rahbar"
password = "password"
site_url = "https://plp.irbroker.com/index.do"

# something before scrape

# initialize driver
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(options=chrome_options)
driver.get(site_url)

# wait until both input elements will be on screen
username_element = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='j_username']")))
password_element = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='j_password']")))
# clear them (not important is site does not use start value)
username_element.clear()
password_element.clear()
# set values
username_element.send_keys(username)
password_element.send_keys(password)
# wait until login button will be clickable and then press it
log_in = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "input[type='submit'][name='B1']"))).click()

# and then thing after scrape, good luck with scrape btw