pythonselenium-webdriverweb-scraping

How to click and loop through html with Selenium Webdriver/Python with no hyperlink


I'm new to scraping and am trying to get the available apartments and their accompanying information from this website (https://sightmap.com/embed/rx1p8y12vd6). Logically, I'm trying to have selenium click through each floor and then pull the dynamic information from each floor (Apt Number, Rent, etc.). The carousel has all the floors in the below format:

<li id="floor-item-42349" role="option" aria-selected="false" aria-label="Floor 21 selected, 1 of 19 stories. 2 Units available." tabindex="-1" class="css-1ge3v7g-itemStyles"><span class="primary-font css-1an6fko-floorStyles">21</span><span class="css-1cygnfi-countStyles"><span class="css-1gmg8q9-numberStyles">2</span><span class="css-1c0jwqn-unitLabelStyles">Units</span></span></li>

Previously, I've seen solutions which have a href to click on which makes it possible, but I don't see where this could be. I'm not familiar with how aria-labels work so maybe this is the key?

Below is what I am working with currently but the click doesn't seem to work.

floors = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "#floor-vertical-select > li ")))
 for floor in floors:
     if floor.text != "0":
         print("Floor: " + floor.text)
         floor.click()

Solution

  • We meet again... :)

    This one is the same basic concept as the last one, just needed updated locators.

    1. Loop through the floors across the left of the page.
    2. If the number of units available is not 0, then print the floor and available unit count, then click it.
    3. Loop through each available unit in the right panel and print it.

    The working code is below.

    from selenium import webdriver
    from selenium.webdriver.support.wait import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    
    #url = f'file:///C:/Users/jbcro/Desktop/sample.html'
    url = 'https://sightmap.com/embed/rx1p8y12vd6'
    driver = webdriver.Chrome()
    driver.maximize_window()
    driver.get(url)
    
    wait = WebDriverWait(driver, 10)
    
    floors = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "#floor-vertical-select span[class*='numberStyles']")))
    for floor in floors:
        if floor.text != "0":
            print("Floor: " + driver.find_element(By.CSS_SELECTOR, "#floor-vertical-select span[class*='floorStyles']").text + ", Available units: " + floor.text)
            floor.click()
            for unit in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "ul[data-testid='unit-list'] li"))):
                print(unit.text)
                print("")
            print("")