I'm trying to scrape shoe sizes from a website using Python and BeautifulSoup. The shoe sizes are located in a div with the class "swatch-option text." I've already managed to extract other information like shoe names and prices, but I'm having trouble getting the shoe sizes.
Here is my current code:
import requests
from bs4 import BeautifulSoup
# Define the URL to scrape
url = "https://www.fuel.com.gr/el/catalogsearch/result/index/?gender=5431%2C5432&q=Dunk"
# Fetch the HTML content of the page
response = requests.get(url)
html = response.text
# Parse the HTML content
soup = BeautifulSoup(html, 'html.parser')
# Find all <li> elements with class "item product product-item"
shoe_items = soup.find_all('li', class_="item product product-item")
for shoe_item in shoe_items:
shoe_name = shoe_item.find('strong', class_='product name product-item-name').text.strip()
shoe_price = shoe_item.find('span', class_='price').text.strip()
# Check if the 'data-mage-init' script exists before extracting data
data_mage_init = shoe_item.find('script', {'type': 'text/x-magento-init'})
if data_mage_init:
# Fetch the individual product page URL
product_link = data_mage_init['data-mage-init']
print("Name:", shoe_name)
print("Price:", shoe_price)
print("Product Link:", product_link)
# Fetch the individual product page
product_response = requests.get(product_link)
product_html = product_response.text
product_soup = BeautifulSoup(product_html, 'html.parser')
# Extract and print the available shoe sizes using the specified CSS selector
shoe_sizes = [size.text.strip() for size in product_soup.select('#amasty-shopby-product-list .text')]
print("Sizes:", ', '.join(shoe_sizes))
print("\n")
else:
# Handle the case where 'data-mage-init' is not found
print("Name:", shoe_name)
print("Price:", shoe_price)
print("Sizes: No sizes available")
print("\n")
I would like to know how to modify this code to extract shoe sizes from the "swatch-option text" div. Any help or guidance on this issue would be greatly appreciated. Thanks!
To extract sizes, use CSS selector "div.swatch-attribute.size", I checked them using the following template: