raft

Raft heartbeat timeout confusion

I am trying to understand Raft.

I have following conclusion, but I am not confident about it.

  1. when idle, leader node need to send heartbeat message at a fixed interval, which must be less than the heartbeat timeout on follower nodes.

  2. when client send real workload request, leader node send AppendEntry message immediately, which have both workload and heartbeat meaning, so the heartbeat timeout on follower nodes will be reset too.

Am I right ?

Edit: enter image description here from this raft visualization , it says log replication will be sent to followers on the next heartbeat, if heartbeat is sent at fixed interval, this will cause obvious latency. why not to send replication immediately here, what's the reason ?

Edit again:

it seems real world implementation works differently from the visualization above. from here:

go-raft would coalesce appends into heartbeats for simplicity, but etcd/raft never worked like that. Even with a heartbeat coalescing policy, latencies are still lower than the heartbeat interval since an entry can be attached to the next heartbeat shortly before the it is due to fire (on average the latency would be half the interval)

Solution

  • You are right overall: https://raft.github.io/raft.pdf page 4 "Rules for Servers" - AppendEntriesRPC resets the time.

    A raft implementation has to make at least this rule implemented: AppendEntriesRPC must be called for each follower till time out happens.

    This rule has three consequences:

    For clarification: entries batching is pretty much standard behaviour of a raft cluster. Even raft specs call for that. Typically, batch is happening when a leader accepts entries at higher rate when a follower acknowledges them. Btw, batching is happening per follower basis: it is a standard behaviour for two different followers to get different set of records as the diff is calculated per follower.