pythonweb-scrapingbeautifulsoupprettify

How to find an element that prettify doesn't show

I am trying to get the origin of this product with Beautifulsoup. I am trying to select the div in which the product data is ordered but I can't.

Later I tried to obtain other div in the code, one of the firsts in the code, but had the same problem. Then I ran prettify and the div I am searching didn't even appear. How can I get this data?

Here is the code I tried:

import urllib.request
from bs4 import BeautifulSoup

urlpage = 'https://www.esselungaacasa.it/ecommerce/nav/auth/supermercato/home.html?freevisit=true#!/negozio/prodotto/5397031?productCode=417932&productType=GROCERY&menuItemId=300000000002399'
page = urllib.request.urlopen(urlpage)
soup = BeautifulSoup(page, 'html.parser')
results = soup.findAll('div', attrs={'class': 'dettaglio'})

I wish I could get all the content of that div so later I can scrape the paragraphs inside it (the 'Origine' paragraph, specifically). Thank you!

Solution

  • The page makes requests for that content to this url:

    https://www.esselungaacasa.it/ecommerce/resources/auth/displayable/breadcrumbs/300000000002399

    This requires headers authentication which seems to comprise as shown below (tested multiple times) . The values are only valid for less than a couple of minutes so you need to see if you can obtain them from a prior request and dynamically update them.

    The json contains html which you can extract and parse with BeautifulSoup.

    import requests    
headers = {
          'Cookie' : 'JSESSIONID=2GqxcW2JyxJ6JvSj7N6VsySBjG29fv4X4tqyVhkcQCJk012YZrJF!-137423361; rxVisitor=155754270561377S2OAJ7NF3RRVHAGOONVM6J8BTDM9E9; _ga=GA1.3.1899558727.1557542711; _gid=GA1.3.1887185695.1557542711; cc_advertising=yes; dtSa=-; BIGipServerPOOL-produzione20.esselungaacasa.it-HTTP=!t1YtlfoXiajamCWJ/a5rCzj/QGm88V4Qo0VUYPxsnhd0TBgWyp+Vfi6oydBlxU/hJ9i5S7kWGT9W/is=; BIGipServerPOOL-ecom30.webapp.esselungaacasa.it-AEM-HTTP=!EMW5HHM3WmpSfPyJ/a5rCzj/QGm88eK13IPf7jx3ZN2rGHroQLAAMcP+cqfG6pU/IQ0WkgGmjLJMCQ8=; dtCookie=1$343B9EA5CDF2E30CCE04D4415DA0CE8D|bdb705b7939fba60|1; XSRF-ECOM-TOKEN=16B8A78F9DC3F2AFFD0137EA22662C77A098944B2FD6F2F2C27693BD76BAF15C; dtLatC=127; BIGipServerPOOL-ecom30.webapp.esselungaacasa.it-8001=!S2wA3HtVHQfvqreJ/a5rCzj/QGm88ZamTbPAvAabBDwyKXTfVg7cipoMLFPFfqZEc5Cotrd56OEwVA==; _gat_UA-79392629-1=1; dtPC=1$544681956_471h17vCBGNMJLCLJIAOCFOMIEGLEBHHPIFOKNI; rxvt=1557546485823|1557542705627',
           'x-dtpc;' : '1$544433049_580h6vCBGNMJLCLJIAOCFOMIEGLEBHHPIFOKNI',
           'X-XSRF-TOKEN' : '16B8A78F9DC3F2AFFD0137EA22662C77A098944B2FD6F2F2C27693BD76BAF15C'

          }

r = requests.get('https://www.esselungaacasa.it/ecommerce/resources/auth/displayable/detail/5397031/300000000002399', headers = headers)

    You can see an example of the json response here

    Content html is inside the list called informations. Sample shown: