I am trying to scrape a website which has Shadow DOM and iframe. As you can see in the image I can't find any selector for the Shadow DOM so I can get to iframe.
How do I locate Shadow DOM (either CSS Selector, XPATH, etc.)?
The page I am trying to scrape is: scrape.do
P.S. I was aiming for the iframe in this way:
iframe = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, 'cf-chl-widget-3kwf9'))
)
The first page you're seeing is actually the Cloudflare Turnstile CAPTCHA that is blocking you from reaching the page that you actually want to go to. It can be bypassed easily with SeleniumBase UC Mode:
from seleniumbase import Driver
driver = Driver(uc=True)
try:
driver.uc_open_with_reconnect("dashboard.scrape.do/login", 4)
driver.uc_gui_click_captcha()
driver.type("#username", "Hello!")
driver.sleep(3)
finally:
driver.quit()
If running on Linux, you'll definitely need to use the SB() format instead, which includes more functionality and a special virtual display:
from seleniumbase import SB
with SB(uc=True) as sb:
url = "dashboard.scrape.do/login"
sb.uc_open_with_reconnect(url, 4)
sb.uc_gui_click_captcha()
sb.type("#username", "Hello!")
sb.sleep(3)