javascriptpythonselenium-webdriver

activate javascript to render Next page on website using selenium


I am trying to understand how to submit request to generate the next page in a website.

The site displays the first 10 rows of a data table, and then provides a list of page numbers and Previous and Next highlighted text (e.g., Previous 1 2 3 4 Next) at the bottom of the data table. I assume there is a javascript that is executed to render the next page. There is no xpath type '//button[text()=Next"] button on the page. On the page, clicking on the highlighted text Next renders the next page.

I have successfully scraped the data from the first displayed table info, but do not understand how to trigger the site to render the next page. I assume the python selenium command to trigger the next page is using:

driver.execute_script('INSERT JS NEXT PAGE COMMAND HERE')

but do not know how to find the name of the js script. I am not familiar with js

This is a portion of the HTML that is displayed. Does this include the name of a js to use as an argument in a driver.execute_script() command?

            <div class="paginator js-paginator hide">
                <a class="previous js-previous icon-with-text">
                    <span class="text">Previous</span> <svg class="icon" height="16" width="16" focusable="false" aria-hidden="true"><use xlink:href="/static/img/components/icons/16px/ic_caretleft.1e8c2f677d23.svg#icon" /></svg>
                </a>
                <div class="pages js-pages hide-for-mobile"></div>
                <a class="next js-next icon-with-text">
                    <span class="text">Next</span> <svg class="icon" height="16" width="16" focusable="false" aria-hidden="true"><use xlink:href="/static/img/components/icons/16px/ic_caretright.b5b45a745d42.svg#icon" /></svg>
                </a>
            </div>

There are also a sequence of scripts included at the bottom of the html page:

    <script src="/static/js/lib/jquery.min.a09e13ee94d5.js" nonce=""></script>
    <script src="/static/js/lib/jquery.deserialize.min.3db41d3f53d5.js" nonce=""></script>
    <script src="/static/js/lib/lodash.min.68e9b65c3984.js" nonce=""></script>
    <script src="/static/js/lib/d3.min.9fe250ef5e1d.js" nonce=""></script>
    <script src="/static/js/lib/select2.min.aebd21499cab.js" nonce=""></script>
    <script src="/static/js/lib/tipped.9093e3c14aaf.js" nonce=""></script>
    <script src="/static/js/lib/vex.min.c9f77cb11959.js" nonce=""></script>
    <script src="/static/js/lib/jquery.waypoints.min.07c8bc20d684.js" nonce=""></script>
    <script src="/static/js/lib/sticky.min.d3f1ddda5800.js" nonce=""></script>

    <script nonce="">
        window.TTAM.ready(function(exports) {
            exports.utility.GTMImpressionsSingleton();
        });
    </script>
        <script src="/static/js/main.40c23f145e01.js" nonce=""></script>

Solution

  • I don't guess I understand why you are looking for a JS command to trigger the next page when there's apparently an A tag that you can click on to go to the next page. Given the HTML you provided, the CSS selector below should work.

    a.next
    

    The code should be something like

    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.support.wait import WebDriverWait
    
    url = 'https://www.whatever.com'
    driver = webdriver.Chrome()
    driver.maximize_window()
    driver.get(url)
    
    wait = WebDriverWait(driver, 10)
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.next"))).click()