pythonurllibhttp-status-code-429

Python HTTP Error 429 with urllib2


I am using the following code to resolve redirects to return a links final url

def resolve_redirects(url):
    return urllib2.urlopen(url).geturl()

Unfortunately I sometimes get HTTPError: HTTP Error 429: Too Many Requests. What is a good way to combat this? Is the following good or is there a better way.

def resolve_redirects(url):
    try:
        return urllib2.urlopen(url).geturl()
    except HTTPError:
        time.sleep(5)
        return urllib2.urlopen(url).geturl()

Also, what would happen if there is an exception in the except block?


Solution

  • It would be better to make sure the HTTP code is actually 429 before re-trying.

    That can be done like this:

    def resolve_redirects(url):
        try:
            return urllib2.urlopen(url).geturl()
        except HTTPError, e:
            if e.code == 429:
                 time.sleep(5);
                 return resolve_redirects(url)
            raise
    

    This will also allow arbitrary numbers of retries (which may or may not be desired).

    https://docs.python.org/2/howto/urllib2.html#httperror