pythonseleniumweb-crawlerpinterest

How to crawl dates in pinterest


I want to crawl by searching for a specific keyword on the pinterest site. However, the date the post was created is not visible when logged in. How can I view or crawl the date the post was created??(I use python)


Solution

  • Yeah, front end normally won't have time of when that posts was created, however I have found a way that you scrape that post's id from DOM(it is available with attribute data-test-id="pin") and make a GET request separately to a URL(using python's request). Here's this example of that URL. This URL will give details about the post in JSON format, which will have time of its creation.

    You can concatenate strings and generate URL like

    protocol+"//"+domain+"/resource/PinResource/get/?source_url=/pin/"+post_id+"/&data={%22options%22:{%22id%22:"+'"'+post_id+"%22,%22field_set_key%22:%22detailed%22,%22is_landing_page%22:false},%22context%22:{}}&_=1523412899257"
    

    You can get protocol and domain like(in python selenium):

    protocol = driver.execute_script("return document.location.protocol")
    
    domain = driver.execute_script("return document.location.hostname")