I'm using BeautifulSoup to scrape a company site job positions (I have permission). The below code is able to run and the output is the url for the job posting, however I want to add in a criteria which has to be true before the url is returned.
Current Code
import requests
from bs4 import BeautifulSoup
base = "http://implementconsultinggroup.com"
url = "http://implementconsultinggroup.com/career/#/1143"
req = requests.get(url).text
soup = BeautifulSoup(req,'html.parser')
links = soup.select("a")
for link in links:
if "career" in link.get("href") and 'COPENHAGEN' in link.text:
res = requests.get(base + link.get("href")).text
soup = BeautifulSoup(res,'html.parser')
title = soup.select_one("h1.page-intro__title").get_text() if
soup.select_one("h1.section__title") else ""
overview = soup.select_one("p.page-intro__longDescription").get_text()
details = soup.select_one("div.rte").get_text()
print(title, link, details)
What I'm trying to achieve
I want to run the above code, but only for the links where the 'Level' is = Graduate do I want to actually show the output. I have written the below, but it does not work.
level = soup.find_all('dd', {'class': 'author'})
if "Graduate" in text
The Website I'm Scraping from
http://implementconsultinggroup.com/career/#/1143
<a href="/career/management-consultants-within-supply-chain-management/" class="box-link">
<h2 class="article__title--tiny" data-searchable-text="">Management consultants within supply chain management</h2>
<p class="article__longDescription" data-searchable-text="">COPENHAGEN • We are looking for bright graduates with a passion for supply chain management and supply chain planning for our planning and execution excellence team.</p>
<div class="styled-link styled-icon">
<span class="icon icon-icon">
<i class="fa fa-chevron-right"></i>
</span>
<span class="icon-text">View Position</span>
</div>
</a>
<div class="small-12 medium-3 columns top-lined">
<dl>
<dt>Position</dt>
<dd class="author">Management Consultant</dd>
<dt>Level</dt>
<dd class="author">Graduate</dd>
<dt>Expertise</dt>
<dd class="author">Operations strategy, Supply chain management</dd>
<dt>Location</dt>
<dd class="author">Copenhagen</dd>
</dl>
</div>
Ideal output
Ideally, I would be able to run the code I have created and it would filter out the positions where the Level != Graduate.
Here you go:
req = requests.get(url).text
soup = BeautifulSoup(req,'html.parser')
for li in soup.find('ul', class_='list-articles list').find_all('li'):
level = li.find_all('dd', {'class': 'author'})[1].get_text()
if "Graduate" in level:
links = li.select("a")
for link in links:
if "career" in link.get("href") and 'COPENHAGEN' in link.text:
res = requests.get(base + link.get("href")).text
soup = BeautifulSoup(res,'html.parser')
title = soup.select_one("h1.page-intro__title").get_text() if soup.select_one("h1.section__title") else ""
overview = soup.select_one("p.page-intro__longDescription").get_text()
details = soup.select_one("div.rte").get_text()
print(title, link, details)