pythonpandasdataframebeautifulsoupespn

Scrape ESPN Current week NFL lines into a pandas dataframe

The following code is only returning the first game. I would like to get all the Week 5 games and lines into a dataframe. Thanks in advance.

import pandas as pd # library for data analysis
import requests # library to handle requests
from bs4 import BeautifulSoup # library to parse HTML documents

# get the response in the form of html
url="https://www.espn.com/nfl/lines"
response=requests.get(url)

# parse data from the html into a beautifulsoup object
soup = BeautifulSoup(response.text,'html.parser')
indiatable=soup.find('section',{'class':"Card"})

df=pd.read_html(str(indiatable))
# convert list to dataframe
df=pd.DataFrame(df[0])
print(df.head())

df

           9:30 AM    REC (ATS)  LINE  OPEN   ML    FPI
0    New York Jets  1-3 (1-3-0)  45.0  43.5  130  42.8%
1  Atlanta Falcons  1-3 (1-3-0)  -2.5  -2.5 -150  56.9%
Out[85]:
9:30 AM REC (ATS)   LINE    OPEN    ML  FPI
0   New York Jets   1-3 (1-3-0) 45.0    43.5    130 42.8%
1   Atlanta Falcons 1-3 (1-3-0) -2.5    -2.5    -150    56.9%
Solution

  • You can use only pandas for this:

    dfs = pd.read_html("https://www.espn.com/nfl/lines")

    dfs - list of dataframes

    To merge on single DataFrame:

    df = pd.concat(dfs)