pythonapache-kafkakafka-pythonconfluent-kafka-python

Can produce message with kafka-python, but can't with confluent-kafka


I try to use confluent-kafka, but can't manage to send message to topic. I did the same with kafka-python, and message sends.

What i am missing?

Thank you!

i use this code

# using confluent-kafka

from confluent_kafka import Producer

producer = Producer({
                    "bootstrap.servers": KAFKA_BOOTSTRAP_SERVERS,
                    "security.protocol": 'SASL_SSL',
                    "sasl.mechanism": 'SCRAM-SHA-512',
                    "sasl.username": secret_creds[KAFKA_VAULT_SECRET_USER_KEY],
                    "sasl.password": secret_creds[KAFKA_VAULT_SECRET_PASSWORD_KEY],
                    "ssl.ca.location": kafka_certificate_path,
                        })

for message in [bytes('confluent_{}'.format(num), 'utf-8') for num in range(10)]:
    producer.produce(topic=KAFKA_TOPIC_NAME, value=message)

producer.flush()

# using kaka-python

from kafka import KafkaProducer

producer_p = KafkaProducer(
    bootstrap_servers=KAFKA_BOOTSTRAP_SERVERS,
    security_protocol='SASL_SSL',
    sasl_mechanism='SCRAM-SHA-512',
    sasl_plain_username=secret_creds[KAFKA_VAULT_SECRET_USER_KEY],
    sasl_plain_password=secret_creds[KAFKA_VAULT_SECRET_PASSWORD_KEY],
        ssl_cafile=kafka_certificate_path,
    )

for message in [bytes('python_{}'.format(num), 'utf-8') for num in range(10)]:
    producer_p.send(KAFKA_TOPIC_NAME, message)

producer_p.flush()

Solution

  • Issue was, that kafka-python accepts bootstrap servers as a list, e.g.

     [
    'kafka1.dev.company.com:9094',
    'kafka2.dev.company.com:9094',
    'kafka3.dev.company.com:9094'
    ]
    

    and confluent-kafka accepts that list as string:

    'kafka1.dev.company.com:9094, kafka2.dev.company.com:9094, kafka3.dev.company.com:9094'