I have three clusters, each with 7 topics and 2 brokers. Each topic is configured with a single partition. Each cluster has one producer and 2 consumers, with the consumers connecting to other clusters. This results in a total of 6 consumers (2 per cluster × 3 clusters) and 3 producers. I have also set up simple ACL authentication using kafka.security.authorizer.AclAuthorizer.
The issue I’m facing is with one of the clusters. When I try to fetch metadata using the following code,
List<PartitionInfo> partitions = consumer.partitionsFor(topic);
it takes more than 40 seconds, and I cannot determine the reason. The exact same code and configuration work perfectly on the other clusters with only 300-500ms. All operations, including broker and ZooKeeper setups, are managed using the Java API.
I can’t understand why this delay occurs only on this specific cluster. Any help in identifying what might be causing this would be greatly appreciated.
To assist with troubleshooting, please consider the following before answering: the configuration, code, and overall setup work perfectly on the two clusters running Linux, while the issue occurs only on Windows. This suggests that the problem may not lie in the code or setup but rather in the host environment.
During my investigation, I ensured the following potential causes are not responsible:
2024-11-23 20:16:54 INFO org.apache.kafka.clients.Metadata - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] Cluster ID: bss9f46kS_qHYiScR08meA). The consumer needs metadata about the topic and partitions to determine fetch positions. i observed also that metadata fetching is slow and with Jprofiler I observed delays on updateFetchPositions but I don't know how to solve itDo you have any other suggestions or ideas about what might be causing the issue?
Here is the consumer config:
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, ip + ":" + KafkaConfiguration.KAFKA_PORT);
props.put(ConsumerConfig.GROUP_ID_CONFIG, groupID);
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, groupID);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, "13107200");
props.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, "536870912");
props.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, "-1");
props.put(ConsumerConfig.SEND_BUFFER_CONFIG, "-1");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, "1");
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, "10");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "60000");
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "600000");
props.put(ConsumerConfig.AUTO_INCLUDE_JMX_REPORTER_CONFIG, "false");
props.put(ConsumerConfig.ENABLE_METRICS_PUSH_CONFIG, "FALSE");
props.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, "1000");
props.put(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG, "300");
props.put(ConsumerConfig.RETRY_BACKOFF_MAX_MS_CONFIG, "7000");
props.put(ConsumerConfig.RECONNECT_BACKOFF_MS_CONFIG, "300");
props.put(ConsumerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG, "7000");
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "11000");
props.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, "8000");
props.put(ConsumerConfig.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_CONFIG, "20000");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, "org.apache.kafka.clients.consumer.StickyAssignor");
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
props.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
props.put(SaslConfigs.SASL_JAAS_CONFIG, "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"" + "consumer" + "-" + position + "-" + current_ip + "\" password=\"consumer-secret\";");
Here is the producer config:
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, KafkaConfiguration.KAFKA_HOST + ":" + KafkaConfiguration.KAFKA_PORT);
props.put(ConsumerConfig.GROUP_ID_CONFIG, KafkaConfiguration.PRODUCER_GROUP_ID + "-" + KafkaConfiguration.KAFKA_HOST);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "1");
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, "56384");
props.put(ProducerConfig.LINGER_MS_CONFIG, "0");
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, "1000");
props.put(ProducerConfig.METADATA_MAX_AGE_CONFIG, "1000");
props.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, "300");
props.put(ProducerConfig.RETRY_BACKOFF_MAX_MS_CONFIG, "7000");
props.put(ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG, "300");
props.put(ProducerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG, "7000");
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, "134217728");
props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, "2024000000");
props.put(ProducerConfig.RECEIVE_BUFFER_CONFIG, "-1");
props.put(ProducerConfig.SEND_BUFFER_CONFIG, "-1");
props.put(ProducerConfig.AUTO_INCLUDE_JMX_REPORTER_CONFIG, "false");
props.put(ProducerConfig.ENABLE_METRICS_PUSH_CONFIG, "false");
props.put(ProducerConfig.RETRIES_CONFIG, "0");
props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, "600000");
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
props.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
props.put(SaslConfigs.SASL_JAAS_CONFIG, "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"producer\" password=\"producer-secret\";");
props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, RoundRobinPartitioner.class.getName());
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);
Lastly here is the kafka broker config:
kafkaProps.setProperty("broker.id", "0");
kafkaProps.setProperty("log.dirs", zkDir);
kafkaProps.setProperty("zookeeper.connection.timeout.ms", "15000");
kafkaProps.setProperty("zookeeper.max.in.flight.requests", "10");
//kafkaProps.setProperty("zookeeper.set.acl","true");
kafkaProps.setProperty("zookeeper.connect", KafkaConfiguration.ZOOKEEPER_HOST + ":" + KafkaConfiguration.ZOOKEEPER_PORT);
kafkaProps.setProperty("offsets.topic.replication.factor", "1");
kafkaProps.setProperty("delete.topic.enable", "true");
kafkaProps.setProperty("num.network.threads", String.valueOf(this.cores));
kafkaProps.setProperty("num.io.threads", String.valueOf(this.cores * 2));
kafkaProps.setProperty("auto.delete.topics.enable", "true");
kafkaProps.setProperty("auto.create.topics.enable", "false");
kafkaProps.setProperty("auto.leader.rebalance.enable", "false");
kafkaProps.setProperty("background.threads", String.valueOf(this.cores / 2));
kafkaProps.setProperty("metrics.reporters", "");
kafkaProps.setProperty("default.api.timeout.ms", "15000");
kafkaProps.setProperty("log.flush.interval.ms", "15000");
kafkaProps.setProperty("max.connections.per.ip,", "250");
kafkaProps.setProperty("remote.fetch.max.wait.ms,", "10");
kafkaProps.setProperty("message.max.bytes", "2024000000");
kafkaProps.setProperty("socket.request.max.bytes", "2024000000");
kafkaProps.setProperty("fetch.max.bytes", "536870912");
kafkaProps.setProperty("group.consumer.heartbeat.interval.ms", "15000");
kafkaProps.setProperty("group.consumer.min.heartbeat.interval.ms", "15000");
kafkaProps.setProperty("group.consumer.max.heartbeat.interval.ms", "45000");
kafkaProps.setProperty("group.consumer.max.session.timeout.ms", "65000");
kafkaProps.setProperty("group.consumer.session.timeout.ms", "60000");
kafkaProps.setProperty("replica.fetch.wait.max.ms", "10");
kafkaProps.setProperty("replica.fetch.max.bytes", "13107200");
kafkaProps.setProperty("auto.include.jmx.reporter", "false");
kafkaProps.setProperty("request.timeout.ms", "25");
kafkaProps.setProperty("socket.receive.buffer.bytes", "-1");
kafkaProps.setProperty("compression.type", "lz4");
kafkaProps.setProperty("socket.send.buffer.bytes", "-1");
kafkaProps.setProperty("socket.connection.setup.timeout.ms", "15000");
kafkaProps.setProperty("group.initial.rebalance.delay.ms", "0");
kafkaProps.setProperty("metadata.max.idle.interval.ms", "1500");
kafkaProps.setProperty("metadata.max.retention.ms", "60480000");
kafkaProps.setProperty("session.timeout.ms", "9000");
kafkaProps.setProperty("heartbeat.interval.ms", "14000");
kafkaProps.setProperty("producer.id.expiration.ms", String.valueOf(ConsensusConfiguration.EPOCH_TRANSITION * 2 * ConsensusConfiguration.CONSENSUS_TIMER));
kafkaProps.setProperty("log.retention.ms", String.valueOf(ConsensusConfiguration.EPOCH_TRANSITION * 2 * ConsensusConfiguration.CONSENSUS_TIMER));
kafkaProps.setProperty("authorizer.class.name", "kafka.security.authorizer.AclAuthorizer");
kafkaProps.setProperty("listeners", "SASL_PLAINTEXT://" + KafkaConfiguration.KAFKA_HOST + ":" + KafkaConfiguration.KAFKA_PORT);
kafkaProps.setProperty("advertised.listeners", "SASL_PLAINTEXT://" + KafkaConfiguration.KAFKA_HOST + ":" + KafkaConfiguration.KAFKA_PORT);
kafkaProps.setProperty("security.inter.broker.protocol", "SASL_PLAINTEXT");
kafkaProps.setProperty("sasl.mechanism.inter.broker.protocol", "PLAIN");
kafkaProps.setProperty("sasl.enabled.mechanisms", "PLAIN");
kafkaProps.setProperty("allow.everyone.if.no.acl.found", "false");
kafkaProps.setProperty("listener.name.sasl_plaintext.plain.sasl.jaas.config", getJaasConfigString());
kafkaProps.setProperty("super.users", getSuperUsers());
Here is an example of kafka-log from the problematic windows machine:
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from __consumer_offsets-13 for epoch 0
2024-11-23 20:16:46 INFO k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Elected as the group coordinator for partition 28 in epoch 0
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from __consumer_offsets-28 for epoch 0
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-3 in 4 milliseconds for epoch 0, of which 0 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-18 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-41 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-10 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-33 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-48 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-19 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-34 in 6 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-4 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-11 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-26 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-49 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-39 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-9 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-24 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-31 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-46 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-1 in 6 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-16 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-2 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-25 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-40 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-47 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-17 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-32 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-37 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-7 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-22 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-29 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-44 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-14 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-23 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-38 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-8 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-45 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-15 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-30 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-0 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-35 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-5 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-20 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-27 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-42 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-12 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-21 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-36 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-6 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-43 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-13 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-28 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Static member with groupInstanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 and unknown member id joins group CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 in Empty state. Created a new member id CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116-58f4af9b-d6c1-4998-ad87-04561bcdeab0 for this member and add to the group.
2024-11-23 20:16:46 INFO k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Preparing to rebalance group CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 in state PreparingRebalance with old generation 0 (__consumer_offsets-7) (reason: Adding new member CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116-58f4af9b-d6c1-4998-ad87-04561bcdeab0 with group instance id Some(CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116); client reason: )
2024-11-23 20:16:46 INFO k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Stabilized group CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 generation 1 (__consumer_offsets-7) with 1 members
2024-11-23 20:16:46 INFO k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Assignment received from leader CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116-58f4af9b-d6c1-4998-ad87-04561bcdeab0 for group CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 for generation 1. The group has 1 members, 1 of which are static.
2024-11-23 20:16:54 INFO org.apache.kafka.clients.Metadata - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] Cluster ID: bss9f46kS_qHYiScR08meA
2024-11-23 20:16:54 INFO org.apache.kafka.clients.Metadata - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106] Cluster ID: GYgEyPiCTzGEuSJvs9r9xA
2024-11-23 20:17:04 INFO o.a.k.c.c.i.ClassicKafkaConsumer - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] Subscribed to topic(s): PREPARE_PHASE, COMMITTEE_PHASE, ANNOUNCE_PHASE, DISPERSE_PHASE2
2024-11-23 20:17:04 INFO o.a.k.c.c.i.ConsumerCoordinator - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] Discovered group coordinator 192.168.1.115:9092 (id: 2147483647 rack: null)
2024-11-23 20:17:04 INFO o.a.k.c.c.i.ClassicKafkaConsumer - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106] Subscribed to topic(s): PREPARE_PHASE, COMMITTEE_PHASE, ANNOUNCE_PHASE, DISPERSE_PHASE2
2024-11-23 20:17:04 INFO o.a.k.c.c.i.ConsumerCoordinator - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106] Discovered group coordinator 192.168.1.116:9092 (id: 2147483647 rack: null)
2024-11-23 20:17:13 INFO o.a.k.c.c.i.ConsumerCoordinator - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] (Re-)joining group
Consumer 192.168.1.115 2 started and subscribed to topics: [PREPARE_PHASE, COMMITTEE_PHASE, ANNOUNCE_PHASE, DISPERSE_PHASE2]
2024-11-23 20:17:13 INFO o.a.k.c.c.i.ConsumerCoordinator - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106] (Re-)joining group
Consumer 192.168.1.116 1 started and subscribed to topics: [PREPARE_PHASE, COMMITTEE_PHASE, ANNOUNCE_PHASE, DISPERSE_PHASE2]
All tasks completed
Finally, I found the answer to my problem. It was very tough to figure out what was really happening. I struggled a lot, and it consumed me hundreds of hours of searching and debugging. The issue was not with the Kafka configuration or API, but a network issue. Specifically, the resolution of the hostname for the IP address of the Ubuntu VM was local, so Windows couldn't resolve the correct hostname from the /etc/hosts file. Kafka handles the exception, but the method runs until a timeout occurs, causing the delay in fetching topic metadata.
Here are the detailed steps to quickly fix the problem:
Make sure ipv4 is enabled or use it programmatically
-Djava.net.preferIPv4Stack=true
Steps to Change Hostname:
Open Terminal: Press Ctrl + Alt + T to open the terminal. Check Current Hostname: To see the current hostname, type:
hostnamectl
This will display the current hostname along with other system information. Set New Hostname: To change the hostname, use the following command:
sudo hostnamectl set-hostname myvmhostname
Replace new-hostname with the desired hostname.
Verify Change: To confirm the change, run:
sh hostnamectl
You should see the new hostname listed under the "Static hostname" section.
Go to windows and do the following: Open the Hosts File: In Notepad, click on File > Open.
Navigate to C:\Windows\System32\drivers\etc.
Select the hosts file. If you don't see it, make sure to change the file type filter to "All Files" at the bottom right1.
Add the IP Address and Hostname:
At the end of the file, add a new line with the IP address of your virtual machine followed by the hostname you want to map. For example:
192.168.1.100 myvmhostname
Save the changes by pressing Ctrl + S or clicking on File > Save
then run the following command in the terminal:
ipconfig /flushdns