pythonselenium-webdriveryahoo-fantasysports-api

Scraping Data from Multiple Pages using Selenium


I am trying to scrape ADP data from Yahoo Fantasy Football and am struggling to get my scraper to move past the initial page (containing 30 players).

I keep receiving a NoSuchElementException when trying to click onto the next page. Any help the experts here could provide would be greatly appreciated.

Below is the code I have been using:

Player_Name = []
Position_Team = []
Draft_Position = []

url = 'https://football.fantasysports.yahoo.com/f1/draftanalysis?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuYmluZy5jb20v&guce_referrer_sig=AQAAAGyHhLSemb5ibj1nJvdsNbQg14lWsna6Y9rAs7DlZ8-d_QA5SkDx_7bjMHMhdGMkQMOLR9F8a56RJ3Xq2zeuUu7QCCivRyHB3FAKv9Oi54Nk01Aev3KFisakR7REUsv42FuWxEKthpGw-BScwgVqzUdRc0a6oATq71C7j85X1C16'

driver.get(url)
time.sleep(1)

for i in range(3):
    players = driver.find_elements(by='xpath', value='//tr[contains(@data-tst, "table-row")]')
delay = 20
      WebDriverWait(driver,delay).until(EC.presence_of_element_located((By.XPATH,'//tr[contains(@data-tst, "table-row")]')))
    for player in players:
        Player_Name.append(player.find_element(by='xpath', value ='.//div[contains(@data-tst, "player-name")]').text)
        Position_Team.append(player.find_element(by='xpath', value = './/div[contains(@class, "C($text-secondary)")]').text)
        Draft_Position.append(player.find_element(by='xpath', value ='.//div[contains(@class, "Ta(c)")]').text)

        next = driver.find_element(by = 'xpath', value = '//*[@id="yui_3_18_1_1_1724006094920_810"]')
        next.click()

Solution

  • Try this:

    next = driver.find_element(By.CSS_SELECTOR, value = 'button[role="presentation"]:last-of-type')
    

    Your selector doesn't seem to work because it seems like Yahoo randomizes the button's ID whenever the page is loaded.