pythonweb-scrapingbeautifulsoupcsvwriter

Beautifulsoup/Writer returns an empty cell when exported to a CSV

I'm scraping a website to get the name, birth + death dates, and the name of the cemetery someone is buried in. For the most part, it is working quite well; however, when I exported the text to a CSV, I noticed that there's a blank cell inserted in the name column after each page. I have a feeling this is probably related to the loop rather than an html tag, but I'm still learning. Any advice is welcome! Thanks everyone

Here's an example of the problem in excel

from dataclasses import replace
import requests
from bs4 import BeautifulSoup
import csv 

api = 'https://www.findagrave.com/memorial/search?'
name = 'firstname=&middlename=&lastname='
years = 'birthyear=&birthyearfilter=&deathyear=&deathyearfilter='
place = 'location=Yulee%2C+Nassau+County%2C+Florida%2C+United+States+of+America&locationId=city_28711'
memorialid = 'memorialid=&mcid='
linkname = 'linkedToName='
daterange = 'datefilter='
plotnum = 'orderby=r&plot='
page = 'page='
url = api + name + "&" + years + "&" + place + "&" + memorialid + "&" + linkname + "&" + daterange + "&" + plotnum + '&' + page


for page_no in range(1,93): 
   url_final = url + str(page_no)
   page = requests.get(url_final, headers = headers)

   #print(page)
   soup = BeautifulSoup(page.content, "html.parser")
   graves = soup.find_all('div', {'class':'memorial-item py-1'})
   #print(graves)

   

   #Getting the Names 
   grave_name = soup.find_all('h2', {'class':'name-grave'}) 

   #Dates
   dates = soup.find_all('b', {'class':'birthDeathDates'})

   #Graveyard Name
   grave_yard = soup.find_all('button', {'role': 'link'})
   #print(grave_yard)

   dataset = [(x.text, y.text, z.text) for x,y,z in zip(grave_name, dates, grave_yard)]
   with open('Fernandiabeach3.csv', 'a',) as csvfile:
    writer = csv.writer(csvfile)
    writer.writerows(dataset)

I've tried to see if there were any similar tags happening at the beginning of each new page, but I couldn't find anything that stood out.

Solution

  • Change

    with open('Fernandiabeach3.csv', 'a',) as csvfile:

    to

    with open('Fernandiabeach3.csv', 'a', newline='') as csvfile:

    In simple terms:

    since you haven't defined how the new line should be it was adding rows

    In technical terms:

    The csv.writer module directly controls line endings and writes \r\n into the file directly. In Python 3 the file must be opened in untranslated text mode with the parameters 'w', newline='' (empty string) or it will write \r\r\n on Windows, where the default text mode will translate each \n into \r\n.

    Hope this helps. Happy Coding :)