I've a micro-service deployed on multiple servers that has two main sources of data:
Events received constantly (24/7/365) at high volume (100-1000 event/sec)
Once-a-day operation which might takes a moment to finish
I want to run this once-a-day processing in exclusive mode: pause the event processing, run the once-a-day task, and then resume the event processing. I already have a way to start the once-a-day operation correctly but I still have to implement locking between 1 and 2 to ensure exclusivity.
Most ZooKeeper recipes that I found require a write operation for every processed event e.g. read lock acquisition with InterProcessReadWriteLock
or counter increase with DistributedAtomicLong
. Since once-a-day opeartions happens infrequenty this per-event overhead seems wasteful.
Is there a ZooKeeper/Curator receipt that's optimized for such a use case?
I thought about following but I'm not 100% sure if that's the right approach (and how to implement point 2 below):
/exclusive
path in ZooKeeper/exclusive
exists. If it's there stop processing until /exclusive
path is removed/exclusive
path What about this?
Each event processor must:
Once this is set there's no additional ZooKeeper activity while it's all running.
When the once a day operation is ready to run:
There's a huge caveat with this, however, and that's what can happen with JVM pauses. Please also read this Tech Note for important edge cases.