I am trying to fetch an element from html page which is hidden deep in multiple shadow roots with selenium
shadow_element = driver.execute_script("return document.querySelector('w3m-modal').shadowRoot.querySelector('div').querySelector('div')")
print(shadow_element.get_attribute('innerHTML'))
Code above reaches a div on the path to my target element and prints out inner html of this div. First line of its inner html is:
<!--?lit$507756472$-->
Which is exactly what my code prints out. But after this line there are 2 more html elements which are ignored by selenium. How do I reach elements below the comment?
HTML:
Below is html of w3m-modal (this is the first element I am selecting with querySelector). What I need to get is <div class="w3m-card">
<w3m-modal>
#shadow-root (open)
<!---->
<w3m-explorer-context></w3m-explorer-context>
<w3m-theme-context></w3m-theme-context>
<w3m-wc-connection-context>
</w3m-wc-connection-context>
<w3m-account-context>
</w3m-account-context>
<w3m-network-context>
</w3m-network-context>
<div id="w3m-modal" role="alertdialog" aria-modal="true" class="w3m-overlay w3m-active" style="opacity: 1;">
<div class="w3m-container" tabindex="0" style="transform: scale(var(--motion-scale)); --motion-scale: 1;">
<!--?lit$656149617$-->
<w3m-modal-backcard></w3m-modal-backcard>
<div class="w3m-card">
<w3m-modal-router></w3m-modal-router>
<w3m-modal-toast></w3m-modal-toast>
</div>
</div>
</div>
</w3m-modal>
To extract the HTML within the #shadow-root (open) you need to induce WebDriverWait for the visibility_of(element)
and you can use the following locator strategy:
print(WebDriverWait(driver, 20).until(EC.visibility_of((driver.execute_script("return document.querySelector('w3m-modal').shadowRoot.querySelector('div.w3m-container')")))).get_attribute('innerHTML'))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC