Leader Election with Leader Latch

Im trying to implement ZK Leader Election using the Leader Latch - Curator framework. For the purposes of this discussion assume that my PCF microservices are spread across 10 datacenters, across timezones and a scheduler kicks off some tasks at a fixed interval. At any time only one instance needs to be the leader to run the scheduled job.

I tried to implement a custom scheduler using ephemereal nodes, but the zookeeper nodes throw a new issue everytime. Instead of re-inventing the wheel I decided to use the curator framework for resolving this issue. The code below copied verbatim with some modifications from https://github.com/yiming187/curator-example/blob/master/src/main/java/com/ctrip/zk/curator/example/LeaderLatchExample.java

CuratorFramework client = null;
LeaderLatch example = null;
try {
  client = CuratorFrameworkFactory.newClient(zkConnString, new ExponentialBackoffRetry(1000, 3));
  example = new LeaderLatch(client, PATH, "Client #"+ i);
  client.start();
  example.start();
  if (example.hasLeadership()) {
    //Since I'm the leader finish processing
  }
  else {
     LOGGER.info("Not elected leader, democracy is overrated !!!");
  }

}
catch(Exception e) {
  e.printStackTrace();
}
finally {
  System.out.println("Shutting down...");
  CloseableUtils.closeQuietly(example);
  CloseableUtils.closeQuietly(client);
}

In doing so, I want to verify :
1) Will a single leader be elected among the 10 instances?
2) My leader processing is idempotent, it does not matter if another leader takes over when the current leader dies. Although I'm guessing by the manner of arranging the code that other potential leaders will have already gone and died when the current leader is interrupted during processing. Am I understanding this correct?
3) Other questions seems to suggest that leadership election may not happen right away in which case, its theoretically possible that the leader may never run? Am I thinking about this correctly. Any inputs will be appreciated

That example code is not ideal, but probably OK for testing purposes. In particular, you'd want to call await(timeout, units). Note, however that that's a blocking call. If you don't want to manage your own threads/locking you can using LeaderSelector instead (see here: https://github.com/apache/curator/tree/master/curator-examples/src/main/java/leader). As to your questions:

1. Yes a single leader will be elected amongst your clients
2. Every client that contends for leadership will eventually get it. But there will only be 1 leader at a time. If a leader crashes, when its client session expires another leader is elected.
3. The way the sample code is structured, leadership will almost certainly never be achieved. Using await(..., ...) or LeaderSelector will solve that.

It's important to understand that all clients that contend for leadership will eventually get it. So, you'll need some other kind of signal that the task has been run. Maybe write a ZNode after it's done so other clients don't repeat the task.

If you're interested, I wrote a complete task scheduling framework for Curator and ZooKeeper that you can use. See here: http://nirmata.github.io/workflow/