apache-kafkaspring-kafka

Kafka: InvalidGroupIdException when manually assigning partition to consumer

I am trying to manually assign partitions to a KafkaListener in Spring boot in order to avoid using GroupManagement and related. Relevant code:

   @KafkaListener(

            topicPartitions = @TopicPartition(
            topic = Globals.INGESTION_KAFKA_TOPIC,
            partitions = {"0", "1", "2", "3", "4", "5", "6", "7"}
         ),
            concurrency = "#{configuration.getQueuePartitionsNumber()}")
    public void listen(@Payload byte[] data,
            @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
            @Header(KafkaHeaders.OFFSET) long currentOffset,
            @Header(KafkaHeaders.RECEIVED_KEY) String key) {

According to documentation this should be enough but through debugging I've discovered that if the AckMode is not set to Record AND Autocommit is false a commit call (for what?) will be done causing errors like:

Caused by: org.apache.kafka.common.errors.InvalidGroupIdException: To use the group management or offset commit APIs, you must provide a valid group.id in the consumer configuration.

What is the minimal way to set up a Spring Boot Kafka Listener that does not use Group Management at all?

Solution

  • The answer was all along in the link:

    You should also set the container’s AckMode to MANUAL to prevent the container from committing offsets for a null consumer group. Starting with version 3.1, the container will automatically coerce the AckMode to MANUAL when manual topic assignment is used with no consumer group.id.

    I don't know why the part in italics doesn't apply, but adding AckMode Manual to the factory config did the trick. ¯_(ツ)_/¯