python pandas web-scraping beautifulsoup

How Can I Scrape Event Links and Contact Information from a Website with Python?

I am trying to scrape event links and contact information from the RaceRoster website (https://raceroster.com/search?q=5k&t=upcoming) using Python, requests, Pandas, and BeautifulSoup. The goal is to extract the Event Name, Event URL, Contact Name, and Email Address for each event and save the data into an Excel file so we can reach out to these events for business development purposes.

However, the script consistently reports that no event links are found on the search results page, despite the links being visible when inspecting the HTML in the browser. Here’s the relevant HTML for the event links from the search results page:

<a href="https://raceroster.com/events/2025/98542/13th-annual-delaware-tech-chocolate-run-5k" 
   target="_blank" 
   rel="noopener noreferrer" 
   class="search-results__card-event-name">
    13th Annual Delaware Tech Chocolate Run 5k
</a>

Steps Taken:

Verified the correct selector for event links:

soup.select("a.search-results__card-event-name")

Checked the response content from the requests.get() call using soup.prettify(). The HTML appears to lack the event links that are visible in the browser, suggesting the content may be loaded dynamically via JavaScript.
Attempted to scrape the data using BeautifulSoup but consistently get:

Found 0 events on the page.
Scraped 0 events.
No contacts were scraped.

What I Need Help With:

How can I handle this JavaScript-loaded content? Is there a way to scrape it directly, or do I need to use a tool like Selenium?
If Selenium is required, how do I properly integrate it with BeautifulSoup for parsing the rendered HTML?

Current Script:

import requests
from bs4 import BeautifulSoup
import pandas as pd

def scrape_event_contacts(base_url, search_url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
    }
    event_contacts = []

    # Fetch the main search page
    print(f"Scraping page: {search_url}")
    response = requests.get(search_url, headers=headers)

    if response.status_code != 200:
        print(f"Failed to fetch page: {search_url}, status code: {response.status_code}")
        return event_contacts

    soup = BeautifulSoup(response.content, "html.parser")
    # Select event links
    event_links = soup.select("a.search-results__card-event-name")


    print(f"Found {len(event_links)} events on the page.")

    for link in event_links:
        event_url = link['href']
        event_name = link.text.strip()  # Extract Event Name

        try:
            print(f"Scraping event: {event_url}")
            event_response = requests.get(event_url, headers=headers)
            if event_response.status_code != 200:
                print(f"Failed to fetch event page: {event_url}, status code: {event_response.status_code}")
                continue

            event_soup = BeautifulSoup(event_response.content, "html.parser")

            # Extract contact name and email
            contact_name = event_soup.find("dd", class_="event-details__contact-list-definition")
            email = event_soup.find("a", href=lambda href: href and "mailto:" in href)

            contact_name_text = contact_name.text.strip() if contact_name else "N/A"
            email_address = email['href'].split("mailto:")[1].split("?")[0] if email else "N/A"

            if contact_name or email:
                print(f"Found contact: {contact_name_text}, email: {email_address}")
                event_contacts.append({
                    "Event Name": event_name,
                    "Event URL": event_url,
                    "Event Contact": contact_name_text,
                    "Email": email_address
                })
            else:
                print(f"No contact information found for {event_url}")
        except Exception as e:
            print(f"Error scraping event {event_url}: {e}")

    print(f"Scraped {len(event_contacts)} events.")
    return event_contacts

def save_to_spreadsheet(data, output_file):
    if not data:
        print("No data to save.")
        return
    df = pd.DataFrame(data)
    df.to_excel(output_file, index=False)
    print(f"Data saved to {output_file}")

if __name__ == "__main__":
    base_url = "https://raceroster.com"
    search_url = "https://raceroster.com/search?q=5k&t=upcoming"
    output_file = "/Users/my_name/Documents/event_contacts.xlsx"

    contact_data = scrape_event_contacts(base_url, search_url)
    if contact_data:
        save_to_spreadsheet(contact_data, output_file)
    else:
        print("No contacts were scraped.")

Expected Outcome:

Extract all event links from the search results page.
Navigate to each event’s detail page.
Scrape the contact name () and email () from the detail page.
Save the results to an Excel file.

Solution

Use the API endpoint to get the data on upcoming events.

Here's how:

import requests
from tabulate import tabulate
import pandas as pd

url = 'https://search.raceroster.com/search?q=5k&t=upcoming'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}

events = requests.get(url,headers=headers).json()['data']

loc_keys = ["address", "city", "country"]

table = [
    [
        event["name"],
        event["url"],
        " ".join([event["location"][key] for key in loc_keys if key in event["location"]])
    ] for event in events
]

columns = ["Name", "URL", "Location"]
print(tabulate(table, headers=columns))

df = pd.DataFrame(table, columns=columns)
df.to_csv('5k_events.csv', index=False, header=True)

This should print:

Name                                         URL                                                                                         Location
-------------------------------------------  ------------------------------------------------------------------------------------------  ----------------------------------------------------------------------------------------------------------------------------
Credit Union Cherry Blossom                  https://raceroster.com/events/2025/72646/credit-union-cherry-blossom                        Washington, D.C. Washington United States
Big Cork Wine Run 5k                         https://raceroster.com/events/2025/98998/big-cork-wine-run-5k                               Big Cork Vineyards, 4236 Main Street, Rohrersville, MD 21779, U.S. Rohrersville United States
3rd Annual #OptOutside Black Friday Fun Run  https://raceroster.com/events/2025/98146/3rd-annual-number-optoutside-black-friday-fun-run  Grain H2O, Summit Harbour Place, Bear, DE, USA Bear United States
Ryan's Race 5K walk Run                      https://raceroster.com/events/2025/97852/ryans-race-5k-walk-run                             Odessa High School, Tony Marchio Drive, Townsend, DE Townsend United States
13th Annual Delaware  Tech Chocolate Run 5k  https://raceroster.com/events/2025/98542/13th-annual-delaware-tech-chocolate-run-5k         Delaware Technical Community College - Charles L. Terry Jr. Campus - Dover, Campus Drive, Dover, DE, USA Dover United States
Builders Dash 5k                             https://raceroster.com/events/2025/99146/builders-dash-5k                                   Rail Haus - Beer Garden, North West Street, Dover, DE Dover United States
The Ivy Scholarship 5k                       https://raceroster.com/events/2025/96874/the-ivy-scholarship-5k                             Hare Pavilion, River Place, Wilmington, DE Wilmington United States
39th Firecracker 5k Run Walk                 https://raceroster.com/events/2025/96907/39th-firecracker-5k-run-walk                       Rockford Tower, Lookout Drive, Wilmington, DE Wilmington United States
24th Annual John D Kelly Logan House 5k      https://raceroster.com/events/2025/97364/24th-annual-john-d-kelly-logan-house-5k            Kelly's Logan House, Delaware Avenue, Wilmington, DE, USA Wilmington United States
2nd Annual Scott Trot 5K                     https://raceroster.com/events/2025/96904/2nd-annual-scott-trot-5k                           American Legion Post 17, American Legion Road, Lewes, DE Lewes United States

Bonus:

To get more events data, just paginate the API with these parameters: l=10&p=1. For example, https://search.raceroster.com/search?q=5k&l=10&p=1&t=upcoming Also, note there's a field in meta -> hits that holds the number of found events. For your query that's 1465.