javaapache-zookeeperapache-curator

Efficient exclusive lock with ZooKeeper for infrequent operations


I've a micro-service deployed on multiple servers that has two main sources of data:

  1. Events received constantly (24/7/365) at high volume (100-1000 event/sec)

  2. Once-a-day operation which might takes a moment to finish

I want to run this once-a-day processing in exclusive mode: pause the event processing, run the once-a-day task, and then resume the event processing. I already have a way to start the once-a-day operation correctly but I still have to implement locking between 1 and 2 to ensure exclusivity.

Most ZooKeeper recipes that I found require a write operation for every processed event e.g. read lock acquisition with InterProcessReadWriteLock or counter increase with DistributedAtomicLong. Since once-a-day opeartions happens infrequenty this per-event overhead seems wasteful.

Is there a ZooKeeper/Curator receipt that's optimized for such a use case?

I thought about following but I'm not 100% sure if that's the right approach (and how to implement point 2 below):

  1. When once-a-day operation starts create new /exclusive path in ZooKeeper
  2. Wait for all in-flight events to finish
  3. Before processing an event check if /exclusive exists. If it's there stop processing until /exclusive path is removed
  4. When once-a-day ends remove the /exclusive path

Solution

  • What about this?

    Each event processor must:

    Once this is set there's no additional ZooKeeper activity while it's all running.

    When the once a day operation is ready to run:

    There's a huge caveat with this, however, and that's what can happen with JVM pauses. Please also read this Tech Note for important edge cases.