apache-kafkakafka-python

Not able to consume messages using Kafka-python consumer SSL and SASL

consumer_SASL = KafkaConsumer(topics,
                              bootstrap_servers=kafkaBrokers,
                              group_id=group,
                              security_protocol='SASL_SSL',
                              sasl_mechanism='PLAIN',
                              ssl_check_hostname=False,
                              ssl_cafile=caRootLocation,
                              ssl_certfile=certLocation,
                              ssl_keyfile=keyLocation,
                              sasl_plain_username=tenable_access_key,
                              sasl_plain_password=password_sasl_key,
                              )

print(consumer_SASL.bootstrap_connected())

for messages in consumer_SASL:
    print(messages)

There is no error but also when I try to check if its connected to bootstrap server with bootstrap_connected() code returns True

With same topic and brokers my Java consumer and producer is working

I was expecting it consume the messages which are being published to this topic. With same topic and brokers my Java consumer and producer is working

Solution

  • This is for someone who is facing same problem. Looks like there is and open issue in kafka-python due to which consumer is getting hung if broker has zstd compression.

    https://github.com/dpkp/kafka-python/issues/2187

    https://github.com/dpkp/kafka-python/issues/2316

    As a workaround you can switch the zstd compression to producer or use snappy or gzip.