I have a producer-consumer implementation that I'm implementing with pyzmq (I took the example from here: I have one producer that produce simultaneously objects to several different types of consumer, where each type of consumer, process this object in a different way.
In addition, I have a manager/supervisor that gets messages regarding the states of the processed objects (e.g. sent_from_producer_to_consumer_1, processed_by_consumer_2, etc), that stores, for each object, all of these states in a dictionary that it's key is the object logical ID and it's values is the list of states.
I want to protect my system from total loss of information in case of a system-failure by recovering it to the latest state it has been at.
So my question is - how can I do that? how can I make sure the manager's objects' states is disk persistent?
I found a package/library called shelve that satisfies my wishes. From the documentation :
A “shelf” is a persistent, dictionary-like object ...