pythongithub

Correctly handle rate limit sleep time in Github


I am writing a script to archive github repos using PyGithub.

    token = os.environ["GITHUB_TOKEN"]

    g = github.Github(auth=github.Auth.Token(token))
    repos = []
    for reponame in repos:
        try:
            repo = g.get_repo(f"{org_name}/{reponame}")
            repo.edit(archived=True)
        except UnknownObjectException:
            pass
        except RateLimitExceededException:
            print("Rate limited")
            time.sleep(g.rate_limiting_resettime - time.time())
            break

Once a rate limit is reached I want to break out of the for loop, sleep until the rate limit is reset and then continue the loop again. How do I do that?

I know I can get the sleep time using g.rate_limiting_resettime - time.time()

https://pygithub.readthedocs.io/en/stable/github.html#github.MainClass.Github.rate_limiting_resettime


Solution

  • Using a for loop here is cumbersome because sometimes you want to retry the same repo if you get rate limited. You also don't want to archive the same repo at the beginning of the list over and over if you restart from scratch. Best here is to keep track of which repo you're processing by index and advance to the next one only once it's processed without error.

    repo_idx = 0
    while repo_idx < len(repos):
        reponame = repos[repo_idx]
        try:
            repo = g.get_repo(f"{org_name}/{reponame}")
            repo.edit(archived=True)
        except UnknownObjectException:
            pass
        except RateLimitExceededException:
            print("Rate limited")
            time.sleep(g.rate_limiting_resettime - time.time())
            continue  # try this repo again from the top
        repo_idx += 1  # no error, so go to the next repo
    

    Consider adding an extra few seconds to your sleep time as a safety buffer from getting rate limited again right away.