pythonweb-scrapinghidden-field

Webscraping for the Hidden Fields

I am web scraping a website: https://apps.ktrade.pk/webterminalv3/SignIn I can fetch HTML from it, but a div with class box-user-id's child elements are not appearing in my scraped html elements; while on inspection element, child elements of box-user-id's are showing.

I have tried it with multiple libraries like selenium, BeautifulSoup, mechanicalsoup etc. Please tell me how can I do it. Thanks in advance.

Solution

  • You can try that code to login

    import requests
from bs4 import BeautifulSoup

session = requests.Session()

# get auth page
auth = session.get('https://apps.ktrade.pk/webterminalv3/SignIn')

# collect csrf token
soup = BeautifulSoup(auth.text, 'html.parser')
csrf_token = soup.find('input', {'name': 'csrfPreventionSalt'})['value']

# create login request
session.post('https://apps.ktrade.pk/webterminalv3/ajax/login', data={
    'username': 'your_username', 'password': 'your_password',
    'csrfPreventionSalt': csrf_token, 'actBrandName': 'KTrade'
})