pythonweb-scrapingtwitternlptwitter-api-v2

snscrape error - twitter scrape crashes after a long time giving '215' error


I got the following error:

https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&q=from%3Ainfobae+since%3A2018-01-01+until%3A2018-12-31&tweet_search_mode=live&count=100&query_source=spelling_expansion_revert_click&cursor=scroll%3AthGAVUV0VFVBaCgLWNpcPlwx0WgICrnczcn_sdEnEV8PFqFYCJehgHREVGQVVMVDUBFdoJFQAA&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel

snscrape.base:4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&q=from%3Ainfobae+since%3A2018-01-01+until%3A2018-12-31&tweet_search_mode=live&count=100&query_source=spelling_expansion_revert_click&cursor=scroll%3AthGAVUV0VFVBaCgLWNpcPlwx0WgICrnczcn_sdEnEV8PFqFYCJehgHREVGQVVMVDUBFdoJFQAA&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel

I used this code:

import snscrape.modules.twitter as sntwitter
import pandas as pd
import re

# Setting variables to be used below
maxTweets = 350000

# Creating list to append tweet data to
tweets_list1 = []
user_list = ['lanacionmas','LANACION','Ambitocom','infobae','clarincom']
# Using TwitterSearchScraper to scrape data and append tweets to list
for user in user_list:
  for i,tweet in enumerate(sntwitter.TwitterSearchScraper('from:' + user +' since:2018-01-01 until:2018-12-31').get_items()):
    if i>maxTweets:
        break

    tweets_list1.append([tweet.date, tweet.id, tweet.content])

print('Done' + user)

Do you know how to fix this error?

Thanks a lot!


Solution

  • Feel like your lucky day, because I created an account here just to reply to you.

    In in this issue, the module creator explains what happened and how to solve the problem, but you just need add top=True in commands that have Twitter**Scraper, or if you are using it from the command line, use the flag --top

    (I just found out that I could have posted as a guest)