pythonbeautifulsoupcraigslist

Python Craiglist Scraping shows empty list

Hi I was using following code to scrape craiglist.

import pandas as pd
import requests
%pylab inline

url_base = 'http://houston.craigslist.org/search/apa'
params = dict(bedrooms=2)
rsp = requests.get(url_base, params=params)

print(rsp.text[:500])
from bs4 import BeautifulSoup as bs4
html = bs4(rsp.text, 'html.parser')
print(html.prettify()[:1000])

everything works fine till above and the output is :-

<!DOCTYPE html>
<html class="no-js">
 <head>
 <title>
  houston apartments / housing rentals  - craigslist
  </title>
<meta content="houston apartments / housing rentals  - craigslist" 
 name="description">
   <meta content="IE=Edge" http-equiv="X-UA-Compatible"/>
   <link href="https://houston.craigslist.org/search/apa" rel="canonical">
   <link href="https://houston.craigslist.org/search/apa?
   format=rss&amp;min_bedrooms=2" rel="alternate" title="RSS feed for 
   craigslist | houston apartments / housing rentals  - craigslist " 
   type="application/rss+xml">
     <link href="https://houston.craigslist.org/search/apa?
 s=120&amp;min_bedrooms=2" rel="next">
      <meta content="width=device-width,initial-scale=1" name="viewport">
       <link href="//www.craigslist.org/styles/cl.css?
 v=a14d0c65f7978c2bbc0d780a3ea7b7be" media="all" rel="stylesheet" 
 type="text/css">
    <link href="//www.craigslist.org/styles/search.css?v=27e1d4246df60da5ffd1146d59a8107e" media="all" rel="stylesheet" type="

It clearly shows that the list is not empty and there are items which i can use. This is use the following code:-

apts = html.find_all('p', attrs={'class': 'row'})
print(len(apts))

The above output of print(len(apts)) is 0.. can anyone please helkp in correcting this code. I do believe there is some change in the craiglist html parser but i dont know how to implement it here.

Thanks

Solution

  • There is no <p> tag with 'row' class instead <p> has 'result-info' class. 

    import requests

url_base = 'http://houston.craigslist.org/search/apa'
params = dict(bedrooms=2)
rsp = requests.get(url_base, params=params)

print(rsp.text[:500])
from bs4 import BeautifulSoup as bs4
html = bs4(rsp.text, 'html.parser')
print(html.prettify()[:1000])

apts = html.find_all('p', attrs={'class': 'result-info'})
print(len(apts))