seleniumgoogle-chromejenkinsselenium-chromedriverrenderer

Selenium build unstable in Jenkins but fine locally

SOLUTIONS POSTED BELOW - TLDR; this dumbs down to using presence_of_element_located with the wrong use case and then not thinking of browser and driver desync.

ISSUE: Issue in which tests pass locally but fail via Jenkins.

Local always passes (windows VM)

Local

Jenkins sometimes passes (windows VM)

Jenkins

Both the local tests and Jenkins tests are run on the same VM (Windows agent). I have followed the advice from these people by adding the following arguments with no change/success:

options = Options()
options.add_experimental_option('excludeSwitches', ['enable-logging'])  #Fixes the problem with console outputing warning "Bluetooth: bluetooth_adapter_winrt.cc:1074 Getting Default Adapter failed."
options.add_argument('--no-sandbox')
options.add_argument("enable-automation")
#options.add_argument("--headless") #CAUSES TIMEOUT ERROR
options.add_argument("--window-size=1920,1080")
options.add_argument("--disable-extensions")
options.add_argument("--dns-prefetch-disable")
options.add_argument("--disable-gpu")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-browser-side-navigation")
options.add_argument("--force-device-scale-factor=1")
prefs = {"download.default_directory" : r"C:\Users\MyUser\Path\Directory\\"} 
options.add_experimental_option("prefs", prefs)
s = Service('chromeDriverPath/chromedriver.exe')
context.driver = webdriver.Chrome(service=s, options=options,desired_capabilities=capa)

The --headless option creates an error:

Timed out receiving message from renderer: 10.000

so I have commented that out.

Question: Why does my local always pass but my Jenkins is unstable even though they run tests on the same VM. How can I fix this?

Background

I have looked at this similar question but no solution from it. Unfortunately I have spent many hours trying to fix this.

Installed Package Versions (PIP)

Package               Version
--------------------- ---------
allure-behave         2.9.45
allure-python-commons 2.9.45
async-generator       1.10
attrs                 21.4.0
behave                1.2.6
cairocffi             1.3.0
CairoSVG              2.5.2
certifi               2022.6.15
cffi                  1.15.1
charset-normalizer    2.1.0
chromedriver          2.24.1
click                 8.1.3
colorama              0.4.5
cryptography          37.0.4
cssselect2            0.6.0
defusedxml            0.7.1
et-xmlfile            1.1.0
h11                   0.13.0
idna                  3.3
lxml                  4.9.1
multipledispatch      0.6.0
numpy                 1.23.1
openpyxl              3.0.10
outcome               1.2.0
pandas                1.4.3
parse                 1.19.0
parse-type            0.6.0
Pillow                9.2.0
pip                   22.0.4
pluggy                1.0.0
pycparser             2.21
pygal                 3.0.0
pyOpenSSL             22.0.0
pypiwin32             223
pyshadow              0.0.4
PySocks               1.7.1
python-dateutil       2.8.2
python-docx           0.8.11
python-dotenv         0.20.0
pytz                  2022.1
pywin32               304
requests              2.28.1
selenium              4.3.0
setuptools            58.1.0
six                   1.16.0
sniffio               1.2.0
sortedcontainers      2.4.0
tinycss2              1.1.1
trio                  0.21.0
trio-websocket        0.9.2
urllib3               1.26.10
webdriver-manager     3.8.1
webencodings          0.5.1
wsproto               1.1.0

Update 08/02/2022 4:11pm

Adding more implicit and explicit wait time fixed many of the flaky tests

Still facing some flaky tests only through Jenkins involving WebDriverWait(context.driver, 180).until(EC.presence_of_element_located((By.CSS_SELECTOR, '{$path}')))

Update 08/03/2022 8:20am SOLUTION

  1. As pointed out by @undected Selenium using presence_of_element_located for this use case may be incorrect. It seems to be the source of the flaky tests. Alternative implementation can be found via the docs. This explains more about why it is bad practice for some scenarios:

One issue was I needed to wait until an element becomes visible instead of just in the DOM. I had to change some presencee_of_element_located() to visibility_of_element_located():

  1. Adding extra time to explicit wait times fixed many flaky tests that were caused from the desync between browser and driver.

Example

Before

WebDriverWait(context.driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, '{$path}')))

After

WebDriverWait(context.driver, 60).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '{$path}')))

Summary

Using the correct expected condition is important. I even had to alter one of mine to element_to_be_clickable to make it not flaky. It was a test involving waiting for a clickable element to pop up anyway. Make sure to read the docs and use the best fit expected condition to avoid errors. Also, look into extending implicit or explicit wait times if its desync errors. Screenshotting upon fail can help you debug this.

Solution

  • There are couple of things you need to address as follows:

    Solution

    Your effective code block can be:

    options = Options()
options.add_argument("--headless")
options.add_argument("--window-size=1920,1080")
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.add_experimental_option("excludeSwitches", ["enable-automation"])
prefs = {"download.default_directory" : r'C:\Users\MyUser\Path\Directory\'} 
options.add_experimental_option("prefs", prefs)
s = Service('chromeDriverPath/chromedriver.exe')
context.driver = webdriver.Chrome(service=s, options=options)