pythonselenium-webdrivershadow-domqueryselectorshadow-root

Selenium: How to fetch elements inside #shadow-root (open) that includes a comment


I am trying to fetch an element from html page which is hidden deep in multiple shadow roots with selenium

shadow_element = driver.execute_script("return document.querySelector('w3m-modal').shadowRoot.querySelector('div').querySelector('div')")
print(shadow_element.get_attribute('innerHTML'))

Code above reaches a div on the path to my target element and prints out inner html of this div. First line of its inner html is:

<!--?lit$507756472$-->

Which is exactly what my code prints out. But after this line there are 2 more html elements which are ignored by selenium. How do I reach elements below the comment?

HTML:

HTML code, comment and elements that follow

Below is html of w3m-modal (this is the first element I am selecting with querySelector). What I need to get is <div class="w3m-card">


<w3m-modal>
#shadow-root (open)
<!---->
<w3m-explorer-context></w3m-explorer-context>
<w3m-theme-context></w3m-theme-context>
<w3m-wc-connection-context>
</w3m-wc-connection-context>
<w3m-account-context>
</w3m-account-context>
<w3m-network-context>
</w3m-network-context>
<div id="w3m-modal" role="alertdialog" aria-modal="true" class="w3m-overlay w3m-active" style="opacity: 1;">
    <div class="w3m-container" tabindex="0" style="transform: scale(var(--motion-scale)); --motion-scale: 1;">
        <!--?lit$656149617$-->
        <w3m-modal-backcard></w3m-modal-backcard>
        <div class="w3m-card">
            <w3m-modal-router></w3m-modal-router>
            <w3m-modal-toast></w3m-modal-toast>
        </div>
    </div>
</div>
</w3m-modal>

Solution

  • To extract the HTML within the #shadow-root (open) you need to induce WebDriverWait for the visibility_of(element) and you can use the following locator strategy:

    print(WebDriverWait(driver, 20).until(EC.visibility_of((driver.execute_script("return document.querySelector('w3m-modal').shadowRoot.querySelector('div.w3m-container')")))).get_attribute('innerHTML'))
    

    Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC