node.jsweb-scrapingtwitter

Fastest way to scrape followers from a Twitter account with Node.js


Due to the limit rate of twitter API I would like to know how could I scrape the number of followers from a Twitter account with Node.js in a very very fast way? Do you think Node.js is the best solution to do it?

I have no idea of how it works but I thought that perhaps scraping to the Twitter mobile version ( example https://mobile.twitter.com/android) would be faster than the computer version.

What do you think? How could I do it?

Thanks!!


Solution

  • Now that the Twitter v1.0 API has been deprecated, you must use oauth signed requests. Which means Twitter knows when you abuse the system.

    Docs on oath signed requests: https://dev.twitter.com/docs/auth/oauth

    Docs on rate limiting: https://dev.twitter.com/docs/rate-limiting/1.1

    Rate limiting applies to the API, not unauthenticated http requests which simply scrape the content.

    If you just want the follower count and not the individual follower ids / details, you can simply scrape the page using request and use jQuery to filter down the results using selectors.

    eg. https://twitter.com/foofighters

    You can run this client side to preview the results:

    $('.js-mini-profile-stats a.js-nav[data-nav=followers] strong').attr('title')