pythonapache-kafkaibm-cloudconfluent-platformmessage-hub

Retrieve messages from a topic on a message hub


I am trying to get messages from a topic on a message hub on bluemix using Confluent Kafka Python. My code is found below, but something is not working. The topic and the message hub is up and running, so there is probably something with the code.

from confluent_kafka import Producer, KafkaError, Consumer

consumer_settings = {
'bootstrap.servers': 'broker-url-here',
'group.id': 'mygroup',
'default.topic.config': {'auto.offset.reset': 'smallest'},
'sasl.mechanisms': 'PLAIN',
'security.protocol': 'ssl',
'sasl.username': 'username-here',
'sasl.password': 'password-here',
}

c = Consumer(**consumer_settings)
c.subscribe(['topic-here'])

running = True

while running:
    msg = c.poll()
    if msg.error():
        print("Error while retrieving message")
        c.close()
        sys.exit(10)
    elif (msg is not None):
        for x in msg:
            print(x)
    else:
        sys.exit(10)

When I run the code, it seems to get stuck at msg = c.poll(). So I guess it is either failing to connect, or failing to retrieve messages. The credentials themselves are correct.


Solution

  • The consume logic look fine but the configuration for the consumer is incorrect.

    We also have a sample app using this client that can easily be started or deployed to Bluemix: https://github.com/ibm-messaging/message-hub-samples/tree/master/kafka-python-console-sample