I'm trying to learn basics of Python & I came across this exercise in a book on how to do Web Scraping. I tried to replicate the code but getting this error- "urllib.error.HTTPError: HTTP Error 406: Not Acceptable".
Is there anything wrong with the code?
I'm using Anaconda/VS Code on Windows 10.
Here's my code:
from urllib import request
from bs4 import BeautifulSoup
page_url = 'https://alansimpson.me/python/scrape_sample.html'
rawpage = request.urlopen(page_url)
soup = BeautifulSoup(rawpage, 'html5lib')
content = soup.article
links_list = []
for link in content.find_all('a'):
try:
url = link.get('href')
img = link.img.get('src')
text = link.span.text
links_list.append({'url' : url, 'img' : img, 'text' : text})
except AttributeError:
pass
And this is the error I'm getting-
Traceback (most recent call last):
File "c:\Users\srika\OneDrive\AIO_Python\scraper.py", line 6, in <module>
rawpage = request.urlopen(page_url)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 214, in urlopen
return opener.open(url, data, timeout)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 523, in open
response = meth(req, response)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 632, in http_response
response = self.parent.error(
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 561, in error
return self._call_chain(*args)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 494, in _call_chain
result = func(*args)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 641, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 406: Not Acceptable
I tried to install 'urllib' but it is already installed. Tried to add exception 'urllib.error.HTTPError', but none of them worked.
How do I solve this? Please help!
Error 406 Not Acceptable status code is an error message that means your website or web application does not support the client's request with a particular protocol.
Add the user agent in headers and try again.
This is the solution for urllib:
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
page_url = 'https://alansimpson.me/python/scrape_sample.html'
req = Request(page_url)
req.add_header('user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36')
rawpage = urlopen(req).read()
soup = BeautifulSoup(rawpage, 'html5lib')
content = soup.article
links_list = []
for link in content.find_all('a'):
try:
url = link.get('href')
img = link.img.get('src')
text = link.span.text
links_list.append({'url' : url, 'img' : img, 'text' : text})
print(links_list)
except AttributeError:
pass
Output:
[{'url': 'http://www.sixthresearcher.com/python-3-reference-cheat-sheet-for-beginners/',
'img': '../datascience/python/basics/basics256.jpg',
'text': 'Basics'},
{'url': 'https://alansimpson.me/datascience/python/beginner/',
'img': '../datascience/python/beginner/beginner256.jpg',
'text': 'Beginner'},
{'url': 'https://alansimpson.me/datascience/python/justbasics/',
'img': '../datascience/python/justbasics/justbasics256.jpg',
'text': 'Just the Basics'},
{'url': 'https://alansimpson.me/datascience/python/cheatography/',
'img': '../datascience/python/cheatography/cheatography256.jpg',
'text': 'Cheatography'},
{'url': 'https://alansimpson.me/datascience/python/dataquest/',
'img': '../datascience/python/dataquest/dataquest256.jpg',
'text': 'Dataquest'},
{'url': 'https://alansimpson.me/datascience/python/essentials/',
'img': '../datascience/python/essentials/essentials256.jpg',
'text': 'Essentials'},
{'url': 'https://alansimpson.me/datascience/python/memento/',
'img': '../datascience/python/memento/memento256.jpg',
'text': 'Memento'},
{'url': 'https://alansimpson.me/datascience/python/syntax/',
'img': '../datascience/python/syntax/syntax256.jpg',
'text': 'Syntax'},
{'url': 'https://alansimpson.me/datascience/python/classes/',
'img': '../datascience/python/classes/classes256.jpg',
'text': 'Classes'},
{'url': 'https://alansimpson.me/datascience/python/dictionaries/',
'img': '../datascience/python/dictionaries/dictionaries256.jpg',
'text': 'Dictionaries'},
{'url': 'https://alansimpson.me/datascience/python/functions/',
'img': '../datascience/python/functions/functions256.jpg',
'text': 'Functions'},
{'url': 'https://alansimpson.me/datascience/python/ifwhile/',
'img': '../datascience/python/ifwhile/ifwhile256.jpg',
'text': 'If & While Loops'},
{'url': 'https://alansimpson.me/datascience/python/lists/',
'img': '../datascience/python/lists/lists256.jpg',
'text': 'Lists'}]