apache-kafkakafka-consumer

what is the difference between client.id and group.instance.id

When configuring a consumer, you can specify:

group.instance.id - "A unique identifier of the consumer instance provided by the end user. Only non-empty strings are permitted. If set, the consumer is treated as a static member, which means that only one instance with this ID is allowed in the consumer group at any time. This can be used in combination with a larger session timeout to avoid group rebalances caused by transient unavailability (e.g. process restarts). If not set, the consumer will join the group as a dynamic member, which is the traditional behavior."

or client.id - "An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging."

for my use case, i needed the exact behavior in the group.instance.id description, but I am wondering why there are two different configuration properties, and it took some time to figure out that client.id is insufficient.

Why are there two?

Solution

  • They are totally different parameters, I've also had to re-read the docs in order to get this clear.

    That means you could have these two values in your consumer A's configuration, for example:

    # Consumer A
client.id = 123
group.instance.id = 1

    The key here is within the static membership paradigm introduced in KIP-345.

    The idea is to avoid the partition changes between consumer threads after a rebalance. For example, if you have three different consumers from the same group.id, and set them unique group.instance.ids, each of those consumers will be assigned the same partitions, always.

    For example, consider three consumers inside the same group.id:

    # Consumer A 
group.instance.id = A

# Consumer B 
group.instance.id = B

# Consumer C
group.instance.id = C

    When reading from a Kafka topic with 9 partitions, they will always hold the same partitions assigned, for example:

    A (0,1,2) - B (3,4,5) - C (6,7,8)

    enter image description here

    What static membership avoids is the partition change caused by the default range assignor, which works by sorting the members in the group and then assigning partition ranges to achieve balance.

    This is well explained in this doc (do not confuse member.id with client.id):

    enter image description here

    1. In red, consumers without group.instance.id have to change partitions with each rebalance due to the assignor.
    2. In green, consumers that did set an instanceId (group.instance.id) becomes static consumers. This static membership makes each consumer "maintain" its initial set of partitions. Even if a rebalance occurs, each consumer always holds the same partitions, avoiding partition changes between consumer threads.

    TL;DR:

    In summary, those parameters are not related and the idea of setting a group.instance.id is to make the consumer "static":

    The idea is summarized as static membership, which in contrary to dynamic membership (the one our system currently uses), is prioritizing "state persistence" over "liveness". The idea of this KIP is to reduce number of rebalances by introducing a new concept called static membership.