pythonweb-scrapingbeautifulsouppython-requestsantiforgerytoken

scrape site with anti forgery token

I'm trying to scrape data from website that uses anti forgery token what i tried to do is sending a get request then finding the key and use it to send a post request i was able to successfully scrape the key but the post request returns error page instead of the data page here is the code i used

note : using the key i scraped to send the request using browser console works fine

import requests
SeatingNo = '330814'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70'
    }

res = requests.get("https://g12.emis.gov.eg/", headers=headers)
soup = BeautifulSoup(res.text, "html.parser")
__RequestVerificationToken = soup.find(type="hidden")['value']
print(__RequestVerificationToken)
data = {'SeatingNo': SeatingNo, '__RequestVerificationToken': __RequestVerificationToken}
res = requests.post("https://g12.emis.gov.eg/", data=data, headers=headers)
soup = BeautifulSoup(res.text, "html.parser")
print(soup)
Solution

  • i was able to solve it using cloudscrapper

    import cloudscraper
scraper = cloudscraper.create_scraper()
SeatingNo = '330814'
res = scraper.get("https://g12.emis.gov.eg/")
soup = BeautifulSoup(res.text, "html.parser")
__RequestVerificationToken = soup.find(type="hidden")['value']
print(__RequestVerificationToken)
data = {'SeatingNo': SeatingNo, '__RequestVerificationToken': __RequestVerificationToken}
res = scraper.post("https://g12.emis.gov.eg/", data=data)
soup = BeautifulSoup(res.text, "html.parser")
print(soup)