pythonexceptionserverinfinite-looprobustness

Robust endless loop for server written in Python

I write a server which handles events and uncaught exceptions during handling the event must not terminate the server.

The server is a single non-threaded python process.

I want to terminate on these errors types:

The list of built in exceptions is long: https://docs.python.org/2/library/exceptions.html

I don't want to re-invent this exception handling, since I guess it was done several times before.

How to proceed?

  1. Have a white-list: A list of exceptions which are ok and processing the next event is the right choice
  2. Have a black-list: A list of exceptions which indicate that terminating the server is the right choice.

Hint: This question is not about running a unix daemon in background. It is not about double fork and not about redirecting stdin/stdout :-)

Solution

  • The top-most exception is BaseException. There are two groups under that:

    Things like Stopiteration, ValueError, TypeError, etc., are all examples of Exception.

    Things like GeneratorExit, SystemExit and KeyboardInterrupt are not descended from Execption.

    So the first step is to catch Exception and not BaseException which will allow you to easily terminate the program. I recommend also catching GeneratorExit as 1) it should never actually be seen unless it is raised manually; 2) you can log it and restart the loop; and 3) it is intended to signal a generator has exited and can be cleaned up, not that the program should exit.

    The next step is to log each exception with enough detail that you have the possibility of figuring out what went wrong (when you later get around to debugging).

    Finally, you have to decide for yourself which, if any, of the Exception derived exceptions you want to terminate on: I would suggest RuntimeError and MemoryError, although you may be able to get around those by simply stopping and restarting your server loop.

    So, really, it's up to you.

    If there is some other error (such as IOError when trying to load a config file) that is serious enough to quit on, then the code responsible for loading the config file should be smart enough to catch that IOError and raise SystemExit instead.

    As far as whitelist/blacklist -- use a black list, as there should only be a handful, if any, Exception-based exceptions that you need to actually terminate the server on.