pythonironwebscraper

Best way to scrape job details from job descriptions


New to web scrapers and I prefer to use Python. Does anyone have any ideas for the easiest way to scrape job descriptions and input them into an excel file? Which scraper would you use?


Solution

  • Libraries I personally use: here

    This is generally the boilerplate code most people probably use to start web scraping:

    import requests
    from bs4 import BeautifulSoup
    import re
    from pprint import pprint
    
    from os.path import dirname, join
    current_dir = dirname(__file__)
    print(current_dir)
    
    code = 0
    
    url_loop = "test.com"
    
    r = (requests.get(url_loop))
    
    error = "The page cannot be displayed because an internal server error has occurred."
    
    soup = BeautifulSoup(r.text, 'html.parser')

    As for using the collected data in excel: Here

    Good luck!