pythonselenium-webdriverbrowser-automation

HttpsConnectionPool Error Selenium: while paste 3000 ids from column in a csv file using Selenium in input field of a URL. Works well for 200 ids

I am using selenium to automate a downloading of a report. for that i need to paste around 3000 ids in a loop for ids around 300 000 into a input field of a webpage and click download button and wait around 40 secs to report to download. And after that click clear button to clear the input field and paste another 3000 values (or ids) into the input field and click download again. Repeat the step till end of all the ids extracted from a column of a dataframe.

The ids are separated by comma.

To do this manually it takes around 2 seconds to paste the values (exactly 3000) and download and then wait for 40 secs and clear and paste another 3000 ids. Repeat the process again.

But while using selenium script after logging into url, closing all the popups and selecting exact option and then while entering (or using input_filed.send_keys(ids) and input_field.send_keys(Keys.ENTER)) there is a timeout error on 120 secs. i dont have 120 secs to wait only for the error to be thrown when it takes only 2 secs manually.

the script works fine for ids around 200 but not 3000. i want 3000 values to be pasted in order the process to be fast.

Kindly provide a solution.

Below is the code which i tried. Please look at the end where is the main problem.

from selenium import webdriver
#you have to create instance wait
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("MY URL")

wait = WebDriverWait(driver, 10)


try: 
    close_button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.toast-close-button")))
    close_button.click()
    print("Popup Closed")
except Exception as e:
    print("No Popup appeared. or popup closed")
    print("Error: ", e)

try:
    username_input = wait.until(EC.presence_of_element_located((By.ID, "username")))
    username_input.send_keys("email_id@something.com")
    print("UserName Entered")
except Exception as e:
    print("Failed to enter Username")
    print("Error: ", e)


try: 
    password_input = wait.until(EC.presence_of_element_located((By.ID, "password")))
    password_input.send_keys("some_password")
    print("Password Entered")
except Exception as e:
    print("Failed to enter Password")
    print("Error: ",e)

try:
    submit_button = wait.until(EC.element_to_be_clickable((By.ID, "loginSubmit"))) 
    submit_button.click()
    print("Submit Button Clicked")
except Exception as e:
    print("Failed to click submit button")
    print("Error: ",e)


ids_option= WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '//*[@id="root"]/div/div/div[2]/div[2]/div[2]/div/div/div[1]/div/div/div/div/div[1]/div'))
)
ids_option.click()

option = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '//div[text()="Option for Id selected"]'))
)
option.click()


input_click_target = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '//*[@id="root"]/div/div/div[2]/div[2]/div[2]/div/div/div[2]/div/div[1]/div/div/div[1]/div[1]'))
)
input_click_target.click()



focused_input = WebDriverWait(driver, 10).until(
    lambda d: d.execute_script("return document.activeElement")
)


from selenium.webdriver.common.keys import Keys
focused_input.send_keys(id_string)        #comma seperated ids
focused_input.send_keys(Keys.ENTER)

Here it throws the error after some 120 secs. If i use only 15 or 200 ids and enter them it works but not for 3000.

please provide a faster solution

I tried the code mentioned above and expecting downloading of individual csv files for 3000 ids each till the end of the ids.

Solution

  • you're trying to paste 3000 IDs quickly into an input box using Selenium, but it hangs or throws HttpsConnectionPool errors (timeouts after 120 sec) only when IDs are many (like 3000), but works fine for small numbers (like 200).

    This is very common because: send_keys is very slow for huge text.The browser lags when pasting huge text via simulated keypresses.The field might revalidate after each keystroke (which is slow for thousands of characters). Network-related operations (XHR/fetch triggered on every change) timeout because the browser is busy or frozen.

    Solution: Use JavaScript injection (bypass slow send_keys) Instead of sending thousands of keystrokes, directly set the value via JavaScript.

    Here’s the improved solution:

    # Instead of send_keys
driver.execute_script("arguments[0].value = arguments[1];", focused_input, id_string)

# If needed, trigger an input event manually
driver.execute_script("""
var input = arguments[0];
var lastValue = input.value;
input.value = arguments[1];
var event = new Event('input', { bubbles: true });
event.simulated = true;
input._valueTracker && input._valueTracker.setValue(lastValue);
input.dispatchEvent(event);
""", focused_input, id_string)

    Explanation:

    Modify this part:

    # Instead of:
# focused_input.send_keys(id_string)
# focused_input.send_keys(Keys.ENTER)

# Use:
driver.execute_script("arguments[0].value = arguments[1];", focused_input, id_string)

# Optionally trigger input event if needed (for apps that listen to 'input' event)
driver.execute_script("""
var input = arguments[0];
var lastValue = input.value;
input.value = arguments[1];
var event = new Event('input', { bubbles: true });
event.simulated = true;
input._valueTracker && input._valueTracker.setValue(lastValue);
input.dispatchEvent(event);
""", focused_input, id_string)

# Then manually click the download button or submit