I am trying to get google search volume index using Pytrends. However, when I use a large set of words, I get the following error:
TooManyRequestsError: The request failed: Google returned a response with code 429
There are similar posts on this issue (for example here). However, none of the solutions in this step work in my case. I thought maybe there is a way to introduce a certain stop in time after several searches? (in total I have 12000 words). When I run the code with a smaller number of words in a list, the code works fine. Below is the code:
from pytrends.request import TrendReq
import pandas as pd
pytrends = TrendReq()
terms_df = pd.read_csv('search_words.csv', sep=',', low_memory=False)
terms = list(terms_df['words'])
def scrape_google(term):
pytrends.build_payload(term, cat=0, timeframe='all', geo='US', gprop='')
trends = pytrends.interest_over_time()
try:
trends = trends.drop(columns=['isPartial'])
except:
pass
return trends
def get_trends(terms):
for i in range(0,len(terms)):
if i == 0:
trends = scrape_google(terms[i:i+1])
else:
trends = pd.concat([trends, scrape_google(terms[i:i+1])], axis=1)
return trends
trends = get_trends(terms)
Alternative way:
from pytrends.request import TrendReq
import pandas as pd
pytrends = TrendReq(hl='en-US', tz=360, timeout=(10, 25))
pytrends.build_payload(terms, cat=0, timeframe='all', geo='US', gprop='')
trends = pytrends.interest_over_time()
trends = trends.drop(columns=['isPartial'])
Here I dont know how to make a loop that will go over a list (terms) and take just 5 words (maximum in pytrends) and that will concatenate pd DF into one.
Here's how you would do 5 at a time. The key is to run your loop in increments of 5, not increments of 1.
def get_trends(terms):
trends = scrape_google(terms[0:5])
for i in range(5,len(terms),5):
trends = pd.concat([trends, scrape_google(terms[i:i+5])], axis=1)
return trends