pythonhtmlweb-scrapingbeautifulsoup

Beautiful Soup findAll doesn't find value


I want to write the prices of the products on this site with the code below with beautifulsoup, but when I write the code, the list returns empty.

import requests as req from bs4 import BeautifulSoup as bs

url = "https://www.migros.com.tr/temel-gida-c-2?sayfa=1"

headers = { 'User-Agent': ( 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) ' 'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36' ) }

urunler = []

rx = req.get(url, headers=headers) sympo = bs(rx.text, 'html.parser') soup = bs(rx.content, 'lxml') print(sympo.findAll('span', {'class': 'amount'}))

print(sympo.findAll('span', {'class': 'amount'}))
[]

Solution

  • The data you see is loaded via Javascript. You can use requests/json module to load it:

    import json
    import requests
    
    
    url = "https://www.migros.com.tr/rest/search/screens/temel-gida-c-2?sayfa=1"
    data = requests.get(url).json()
    
    # uncomment this to print all data:
    # print(json.dumps(data, indent=4))
    
    for p in data["data"]["searchInfo"]["storeProductInfos"]:
        print(
            "{:<35} {:<10} {:<10}".format(
                p["name"], p["regularPrice"], p["salePrice"]
            )
        )
    

    Prints:

    Maydanoz Adet                       445        445       
    Soğan Kuru Dökme Kg                 195        195       
    Migros Havuç Beypazarı Paket Kg     875        750       
    Domates Kg                          1495       1495      
    Kabak Sakız Kg                      1990       1990      
    Dereotu Adet                        930        930       
    Roka Demet                          603        603       
    Salata Kıvırcık Adet                1090       1090      
    Patlıcan Kemer Kg                   1990       1990      
    Soğan Taze Demet                    925        925       
    Hıyar Kg                            1790       1790      
    Domates Salkım Kg                   2290       2290      
    Biber Kırmızı Kg                    2190       2190      
    Brokoli Kg                          3450       3450      
    Atom Salata Adet                    1206       1206      
    Kereviz Kg                          875        875       
    Karnabahar Kg                       1390       1390      
    Ispanak Kg                          1450       1450      
    Patates Taze Kg                     556        556       
    Biber Köy Usulü Kg                  2990       2990      
    Nane Adet                           631        631       
    Biber Sivri Kg                      2690       2690      
    Pırasa Kg                           930        930       
    Lahana Beyaz Kg                     595        595       
    Biber Dolmalık Kg                   2702       2702      
    Domates Şeker 250 G                 837        837       
    Lahana Kırmızı Kg                   1206       1206      
    Patates Ekonomik Boy File Kg        445        445       
    Pancar Kg                           743        743       
    Domates Pembe Kg                    1850       1850