Do you know if there is well designed (state machine based) heartbeat mechanism written in Java as opensource project? Actually state-based is not mandatory for me. I just assume that this kind of design covers various failover case.
After google searching I have found Java DMK. Your responses will be helpful for me to avoid testing all such projects to find the better one.
I could point you to JGroups project. It has decentralized system that has cluster state and heardbeat functionality out of the box.
Also there are famous ZooKeeper project, that also has some of your needs.
Hope it helps.