pythonpython-3.xweb-scrapingpython-requests

Failed to find out the source of a certain portion of a link


I've created a script in python to scrape certain fields from a webpage. When I use this link in the script, it produces all the data in json format and I can parse it accordingly.

import requests

link = 'https://api-emis.kemenag.go.id/v1/institutions/pontren/public/identity/K1hOenRreVRmaWYwSGVzcERWVFpjZz09'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36'
}

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    container = res.json()['results']
    account_bank_holder = container['account_bank_holder']
    organizer_name = container['organizer_name']
    print((account_bank_holder,organizer_name))

However, the problem is that I can't figure out where this portion K1hOenRreVRmaWYwSGVzcERWVFpjZz09 at the end of the link is coming from.

This is how I found the link:

  1. Navigate to this website
  2. Select 2023/2024 Genap
  3. Click the first item in the table: ACEH
  4. Then, click the first item in the new table: ACEH SELATAN
  5. Finally, click the first item in the new table: 500311010057

You should have the link by now, ending with the portion I mentioned above.

Question: Where is the portion similar to this K1hOenRreVRmaWYwSGVzcERWVFpjZz09 at the end of the final link coming from?


Solution

  • So, developer has used aes encryption along with base64 encoding.

    import base64
    from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
    from cryptography.hazmat.backends import default_backend
    from cryptography.hazmat.primitives import padding
    
    def aes_encrypt_cbc(plaintext: str) -> str:
        secret_key = b'a2c36eb2w1em50d6665dc5d61a68b400'
        base64_iv = "ZW1pc0Jhc2U2NElWa2V5cw=="
        iv = base64.b64decode(base64_iv)
        cipher = Cipher(algorithms.AES(secret_key), modes.CBC(iv), backend=default_backend())
        encryptor = cipher.encryptor()
        padder = padding.PKCS7(128).padder()
        padded_plaintext = padder.update(plaintext.encode()) + padder.finalize()
        ciphertext = encryptor.update(padded_plaintext) + encryptor.finalize()
        return base64.b64encode(base64.b64encode(ciphertext).decode().encode()).decode()
    
    institution_id = "175974"
    encrypted_text = aes_encrypt_cbc(institution_id)
    print("Encrypted: ", encrypted_text)
    

    Note: Initialization Vector and secret key may change in future.!