pythonpython-3.xweb-scrapingbeautifulsouptripadvisor

Review scraping form tripadvisor


I am new to web scraping in python3. I want to scrape the reviews of all the hotels in dubai but the problem is I can only scrape the hotel review which I describe in the url. Can anyone show me how I can get all of the hotel reviews without implicitly giving url of each hotel?

import requests
from bs4 import BeautifulSoup


importurl = 'https://www.tripadvisor.com/Hotel_Review-g295424-d302778-Reviews-Roda_Al_Bustan_Dubai_Airport-Dubai_Emirate_of_Dubai.html'
r = requests.get(importurl)
soup = BeautifulSoup(r.content, "lxml")
 resultsoup = soup.find_all("p", {"class" : "partial_entry"})
#save the reviews to a test text file locally
for review in resultsoup:
review_list = review.get_text()
print(review_list)
with open('testreview.txt', 'w') as fid: 
    for review in resultsoup:
        review_list = review.get_text()
        fid.write(review_list)

Solution

  • you should find the index page of all hotel, get all the link into a list, than loop the url list to get comment.

    import bs4, requests
    index_pages = ('http://www.tripadvisor.cn/Hotels-g295424-oa{}-Dubai_Emirate_of_Dubai-Hotels.html#ACCOM_OVERVIEW'.format(i) for i in range(0, 540, 30))
    urls = []
    with requests.session() as s:
        for index in index_pages:
            r = s.get(index)
            soup = bs4.BeautifulSoup(r.text, 'lxml')
            url_list = [i.get('href') for i in soup.select('.property_title')]
            urls.append(url_list)
    

    out:

    len(urls): 540