Following this tutorial https://www.scrapingdog.com/blog/scrape-indeed-using-python/, and running into this error:
Traceback (most recent call last):
File "C:/Users/det-lab/Documents/PycharmProjects/Indeed_webscrape/Indeed_job_data.py", line 23, in <module>
alllitags = allData.find_all("li", {"class":"eu4oa1w0"})
AttributeError: 'NoneType' object has no attribute 'find_all'
Here is my version of the program:
import requests
from bs4 import BeautifulSoup
l = []
o = {}
#declare the target URL and make an HTTP connection to that website.
target_url = 'https://www.indeed.com/jobs?q=data+analyst&l=New+York%2C+NY&from=searchOnHP&vjk=7cfb06a1924a00ef&advn\
=8756188910781422'
head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0",
"Accept-Encoding": "gzip, deflate, br",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Connection": "keep-alive",
"Accept-Language": "en-US,en;q=0.9,lt;q=0.8,et;q=0.7,de;q=0.6",
}
resp = requests.get(target_url, headers=head)
soup = BeautifulSoup(resp.text, 'html.parser')
allData = soup.find("div", {"class": "mosaic-provider-jobcards"})
#iterate over each of these li tags and extract all the data one by one using a for loop.
alllitags = allData.find_all("li", {"class":"eu4oa1w0"})
for i in range(0,len(alllitags)):
try:
o["name-of-the-job"]=alllitags[i].find("a").find("span").text
except:
o["name-of-the-job"] = None
try:
o["name-of-the-company"] = alllitags[i].find("span", {"data-testid":"company-name"}).text
except:
o["name-of-the-company"] = None
try:
o["job-location"] = alllitags[i].find("div", {"data-testid":"text-location"}).text
except:
o["job-location"] = None
try:
o["job-details"] = alllitags[i].find("div", {"class":"jobMetaDataGroup"}).text
except:
o["job-details"] = None
try:
o["pay-range"] = alllitags[i].find("div", {"class":"metadata salary-snippet-container"}).text
except:
o["pay-range"] = None
l.append(o)
o={}
print(l)
I'm aware this is a common error when dealing with webscraping but id like a specific explanation.
Please Add your Cookie value along with the User-Agent header because Indeed is protected by Cloudflare.(I hunt for bugs in Indeed almost every day):
Just added the cookie Key in the header (You should fill the value of yours):
import requests
from bs4 import BeautifulSoup
l = []
o = {}
#declare the target URL and make an HTTP connection to that website.
target_url = 'https://www.indeed.com/jobs?q=data+analyst&l=New+York%2C+NY&from=searchOnHP&vjk=7cfb06a1924a00ef&advn\
=8756188910781422'
head = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0',
'Accept-Encoding': 'gzip, deflate, br',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Connection': 'keep-alive',
'Cookie': '{YOUR WWW.INDEED.COM COOKIE IS HERE}',
'Accept-Language': 'en-US,en;q=0.9,lt;q=0.8,et;q=0.7,de;q=0.6'
}
resp = requests.get(target_url, headers=head)
soup = BeautifulSoup(resp.text, 'html.parser')
allData = soup.find("div", {"class": "mosaic-provider-jobcards"})
#iterate over each of these li tags and extract all the data one by one using a for loop.
alllitags = allData.find_all("li", {"class":"eu4oa1w0"})
for i in range(0,len(alllitags)):
try:
o["name-of-the-job"]=alllitags[i].find("a").find("span").text
except:
o["name-of-the-job"] = None
try:
o["name-of-the-company"] = alllitags[i].find("span", {"data-testid":"company-name"}).text
except:
o["name-of-the-company"] = None
try:
o["job-location"] = alllitags[i].find("div", {"data-testid":"text-location"}).text
except:
o["job-location"] = None
try:
o["job-details"] = alllitags[i].find("div", {"class":"jobMetaDataGroup"}).text
except:
o["job-details"] = None
try:
o["pay-range"] = alllitags[i].find("div", {"class":"metadata salary-snippet-container"}).text
except:
o["pay-range"] = None
l.append(o)
o={}
print(l)
[{'name-of-the-job': 'Business Analyst', 'name-of-the-company': 'NYU Langone Health', 'job-location': 'New York, NY 10016\xa0(Tudor City area)', 'job-details': '\n5 years experience analyzing data and working with business users in a warehouse/data lake environment Ability to translate business requirements into technical…\n·More...View all NYU Langone Health jobs in New York, NY - New York jobsSalary Search: Business Analyst salaries in New York, NYSee popular questions & answers about NYU Langone Health', 'pay-range': None}, {'name-of-the-job': 'Data Science Analyst I-Geriatric and Palliative Medicine-12791-002', 'name-of-the-company': 'Mount Sinai', 'job-location': 'New York, NY 10029', 'job-details': '\nB. Identifies necessary data, data sources and methodologies.\nAdheres to corporate standards for performance metrics, data collection, data integrity, query…\n·More...View all Mount Sinai jobs in New York, NY - New York jobs - Data Scientist jobs in New York, NYSalary Search: Data Science Analyst I-Geriatric and Palliative Medicine-12791-002 salaries in New York, NYSee popular questions & answers about Mount Sinai', 'pay-range': None}, {'name-of-the-job': 'Data Analyst', 'name-of-the-company': 'PACO Group', 'job-location': 'Hybrid work in New York, NY 10038', 'job-details': '\nAbility to identify relevant trends in economic, demographic, and transportation data to inform analyses such as an inflation analysis of the agency’s capital…\n·More...View all PACO Group jobs in New York, NY - New York jobsSalary Search: Data Analyst salaries in New York, NY', 'pay-range': None}, {'name-of-the-job': 'Data Analyst - Patient Experience', 'name-of-the-company': 'Ro', 'job-location': 'New York, NY 10011', 'job-details': 'Typically responds within 1 day\nBuild out data availability and business intelligence solutions for your team.\nLeverage data to shape your team/business’ strategy, providing support and/or…\n·More...View all Ro jobs in New York, NY - New York jobs - Data Analyst jobs in New York, NYSalary Search: Data Analyst - Patient Experience salaries in New York, NYSee popular questions & answers about Ro', 'pay-range': None}, {'name-of-the-job': 'Finance Data Analyst', 'name-of-the-company': 'ACORD Solutions Group', 'job-location': 'New York, NY', 'job-details': '\nThis position requires 3-5 days in the office as coordinated with the MCF global senior Business Manager.\nAnalyze financial data, variances, forecasts and track…\n·More...View all ACORD Solutions Group jobs in New York, NY - New York jobs - Data Analyst jobs in New York, NYSalary Search: Finance Data Analyst salaries in New York, NY', 'pay-range': None}, {'name-of-the-job': None, 'name-of-the-company': None, 'job-location': None, 'job-details': None, 'pay-range': None}, {'name-of-the-job': 'Data Analyst', 'name-of-the-company': 'NYU Langone Health', 'job-location': 'New York, NY 10016\xa0(Tudor City area)', 'job-details': '\nExperience with literature review and data synthesis.\nUpdate and maintain study databases, assist with data analysis.\nOrganize and maintain research records.\n·More...View all NYU Langone Health jobs in New York, NY - New York jobsSalary Search: Data Analyst salaries in New York, NYSee popular questions & answers about NYU Langone Health', 'pay-range': None}, {'name-of-the-job': 'Analyst, Citywide Data, RPSG', 'name-of-the-company': 'New York City Department of Education', 'job-location': 'Manhattan, NY', 'job-details': '\nImplement data validation and cleaning processes to maintain data quality standards.\nYou will work closely with cross-functional teams and data partners to…\n·More...View all New York City Department of Education jobs in Manhattan, NY - Manhattan jobs - Data Analyst jobs in Manhattan, NYSalary Search: Analyst, Citywide Data, RPSG salaries in Manhattan, NYSee popular questions & answers about New York City Department of Education', 'pay-range': None}, {'name-of-the-job': 'Senior Risk Data Analyst', 'name-of-the-company': 'Kirkland and Ellis', 'job-location': 'Hybrid work in New York, NY 10022', 'job-details': '\nMaintain data integrity through quality checks, data validation, and proper data storage.\nClean and preprocess data, ensuring data accuracy and completeness.\n·More...View all Kirkland and Ellis jobs in New York, NY - New York jobs - Senior Risk Analyst jobs in New York, NYSalary Search: Senior Risk Data Analyst salaries in New York, NYSee popular questions & answers about Kirkland and Ellis', 'pay-range': None}, {'name-of-the-job': 'Enterprise Data Analyst', 'name-of-the-company': 'New York City Department of Education', 'job-location': 'Brooklyn, NY 11201', 'job-details': '\nLeads the development of internal and external reports and responds to ad hoc data requests.\nStrong attention to detail and extremely well organized; aptitude…\n·More...View all New York City Department of Education jobs in Brooklyn, NY - Brooklyn jobs - Data Analyst jobs in Brooklyn, NYSalary Search: Enterprise Data Analyst salaries in Brooklyn, NYSee popular questions & answers about New York City Department of Education', 'pay-range': None}, {'name-of-the-job': 'RPIE Data Analyst & Program Assessment', 'name-of-the-company': 'The Cooper Union for the Advancement of Science...', 'job-location': 'New York, NY 10003\xa0(NoHo area)', 'job-details': '\nCollect and analyze participant data.\nExperience developing data tracking systems.\nRPIE’s Data Analyst role is responsible for tracking, collecting, and…\n·More...View all The Cooper Union for the Advancement of Science and Art jobs in New York, NY - New York jobs - Data Analyst jobs in New York, NYSalary Search: RPIE Data Analyst & Program Assessment salaries in New York, NY', 'pay-range': None}, {'name-of-the-job': None, 'name-of-the-company': None, 'job-location': None, 'job-details': None, 'pay-range': None}, {'name-of-the-job': 'Data Analyst', 'name-of-the-company': 'Hill Data Management', 'job-location': 'New York, NY', 'job-details': 'Work Location NYC-Remote Expected Duration December 2028 Type Part-time Contract Experience 5-7 years Education Level Bachelors or relevant experience Salary…·More...View all Hill Data Management jobs in New York, NY - New York jobsSalary Search: Data Analyst salaries in New York, NY', 'pay-range': None}, {'name-of-the-job': 'Business Analyst', 'name-of-the-company': 'NYU Langone', 'job-location': 'New York, NY 10016\xa0(Tudor City area)', 'job-details': '\n5 years experience analyzing data and working with business users in a warehouse/data lake environment Ability to translate business requirements into technical…\n·More...View all NYU Langone jobs in New York, NY - New York jobsSalary Search: Business Analyst salaries in New York, NYSee popular questions & answers about NYU Langone', 'pay-range': None}, {'name-of-the-job': 'BP Process and Data Analyst II - NY, NJ, Or PA', 'name-of-the-company': 'Visions Federal Credit Union', 'job-location': 'Remote in New York, NY', 'job-details': '\nBachelor’s degree and 1-3 years of related process improvement and data analysis experience is expected, 4-6 years of relevant experience are required without a…\n·More...View all Visions Federal Credit Union jobs in New York, NY - New York jobs - Senior Data Analyst jobs in New York, NYSalary Search: BP Process and Data Analyst II - NY, NJ, Or PA salaries in New York, NYSee popular questions & answers about Visions Federal Credit Union', 'pay-range': None}, {'name-of-the-job': 'Data Analyst', 'name-of-the-company': 'NYU Langone', 'job-location': 'New York, NY 10016\xa0(Tudor City area)', 'job-details': '\nExperience with literature review and data synthesis.\nUpdate and maintain study databases, assist with data analysis.\nOrganize and maintain research records.\n·More...View all NYU Langone jobs in New York, NY - New York jobsSalary Search: Data Analyst salaries in New York, NYSee popular questions & answers about NYU Langone', 'pay-range': None}, {'name-of-the-job': 'Senior Data Analyst', 'name-of-the-company': 'Guy Carpenter', 'job-location': 'Hybrid work in New York, NY 10036', 'job-details': '\nInterpret and analyze company financial data to identify trends and opportunities in observed data.\nBe a trusted leader and resource for maintaining and…\n·More...View all Guy Carpenter jobs in New York, NY - New York jobsSalary Search: Senior Data Analyst salaries in New York, NYSee popular questions & answers about Guy Carpenter', 'pay-range': None}, {'name-of-the-job': None, 'name-of-the-company': None, 'job-location': None, 'job-details': None, 'pay-range': None}]