I have some code to open a search url for a site, click the first result, and click a button. This all works fine until I try to use headless Chrome.
Code (working -> not using headless Chrome):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
browser = webdriver.Chrome()
browser.get("https://www.google.com/search?q=chatgpt")
f=browser.find_elements(By.TAG_NAME, "h3")
f[0].click()
button = browser.find_elements(By.TAG_NAME, 'button')
button[0].click()
print("Page title was '{}'".format(browser.title))
input()
Output (which is what I want):
DevTools listening on ws://127.0.0.1:58358/devtools/browser/e8f33ba5-9423-4900-9d36-035615599b61
Page title was 'ChatGPT'
[6464:11928:0914/044721.426:ERROR:device_event_log_impl.cc(225)] [04:47:21.426] USB: usb_service_win.cc:415 Could not read device interface GUIDs: The system cannot find the file specified. (0x2)
Code (not working -> using headless Chrome):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
browser = webdriver.Chrome(options=chrome_options)
browser.get("https://www.google.com/search?q=chatgpt")
f=browser.find_elements(By.TAG_NAME, "h3")
f[0].click()
button = browser.find_elements(By.TAG_NAME, 'button')
button[0].click()
print("Page title was '{}'".format(browser.title))
input()
Error:
DevTools listening on ws://127.0.0.1:58289/devtools/browser/e4c33772-2e8c-4060-96b6-6aa730ef53c2
[0914/044642.023:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'unload'.", source: (0)
[0914/044644.747:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'unload'.", source: (0)
[0914/044645.283:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'browsing-topics'.", source: (0)
[0914/044645.284:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest-cohort'.", source: (0)
[0914/044645.435:INFO:CONSOLE(0)] "Refused to execute script from 'https://chat.openai.com/cdn-cgi/challenge-platform/h/g/scripts/alpha/invisible.js?ts=1694649600' because its MIME type ('') is not executable, and strict MIME type checking is enabled.", source: about:blank (0)
[0914/044646.033:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'browsing-topics'.", source: (0)
[0914/044646.033:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest-cohort'.", source: (0)
Traceback (most recent call last):
File "d:\RapidAPI\willitworkorwillittwerk.py", line 14, in <module>
button[0].click()
~~~~~~^^^
IndexError: list index out of range
I tried google, youtube, chatgpt, and tiktok. Most sites are giving me the same thing, but example.com worked for some reason.
This is my first time trying to use headless mode. I asked chatgpt and it said this isn't intended behavior, so if anyone has any information, that would be great.
I also tried undetected_chromedriver, which I might just use, but it gave similar errors.
Chrome Version: 117.0.5938.63 Webdriver Version: 117.0.5938.62 Selenium Version: 4.12.0 i added --headless==new it removed the cloudflare errors but im still getting the out of index error but when its not headless i dont get it
You have to use the new headless mode to get the same results as normal mode:
chrome_options.add_argument("--headless=new")
You'll see that if you search for --headless=new in the Selenium documentation on https://www.selenium.dev/documentation/webdriver/browsers/chrome/
If you're still having issues, try SeleniumBase UC Mode after pip install seleniumbase:
from seleniumbase import Driver
try:
driver = Driver(uc=True, headless2=True)
driver.get("https://www.google.com/search?q=chatgpt")
driver.click("h3")
driver.click("button")
print(driver.title)
finally:
driver.quit()
Here's the result after running with python:
ChatGPT