pythonrequesturllib

Retry Request When Its Failed


I Programmed this code to get website content but there is a problem when connection is filed the program stop and dont try to reconnect

            url= 'https://website.com'
            def get_page_content(url, head):
              """
              Function to get the page content
              """
              req = Request(url, headers=head)
              return urlopen(req)
            head = {
              'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36',
              'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
              'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
              'Accept-Encoding': 'none',
              'Accept-Language': 'en-US,en;q=0.8',
              'Connection': 'keep-alive',
              'refere': 'https://example.com',
              'cookie': """your cookie value ( you can get that from your web page) """
            }

            data = get_page_content(url, head).read()

I searched in the internet but didn't find solution.


Solution

  • Python has try, except blocks you can add. Basically you put the code that's failing (specifically the lines that throws errors) into the try section, and then the except portion will contain code that you want to do in the event of an error. You can use this with a flag in a loop to keep trying to connect a certain number of times

    success_flag = False
    while success_flag == False: # keep looping until success
         try: # code in here will attempt to run
              req = Request(url, headers=head)
              page = urlopen(req)
              success_flag = True # mark as successful run if got through the code that may fail
              return page
         except Exception as e: # this code will run in the event of a failure
              print(e) # just print the exception before trying again. 
              # possibly change something before reconnecting depending on error thrown
    

    keep in mind you don't have the traceback posted, so there may be an issue with the code itself. This code will just do as you asked and that is keep trying again. You may need to change something before trying a reconnect