juliablink

How to use Blink to scrape webpages with javascript calling encrypted or obfuscated json?


It is understood from Mr.Avik Sengupta youtube video on NLP, that:
a) HTTP.get(url) cannot be used to scrape web pages that uses javascript to call encrypted or obfuscated json.
b) Blink can to be used in such cases.

I am new to Blink and coulldn't find any useful documentation/exmple in this regard!

Please guide me in using Blink to scrape such webpages.


Solution

    1. It is JavaScript not jsp
    2. If you are new into subject try first Selenium (it should work via PyCall). There are several great tutorials for Selenium and going through them will allow to understand web driver architecture
    3. Once you have experience with Selenium, the Blink documentation at https://juliagizmos.github.io/Blink.jl/latest/communication/ should be quite obvious. You will basically need to navigate to a web page and than use JavaScript to obtain its elements.
    4. Last but not least - for this type of projects always check whether scraping is legal and ethical in your use case.