"Why bare metal?" - It's a company limitation/bureaucracy.
I'm trying to startup a Kafka Cluster on a single machine for the sake of learning before trying to send things to production.
My goal is to setup 1 Controller, 1 Broker,Controller and 1 Broker. I'm editing directly the server.properties file.
Here is the code for the Controller
process.roles=controller
node.id=1
controller.quorum.voters=1@localhost:9193,2@localhost:9293
listeners=CONTROLLER1://localhost:9193
controller.listener.names=CONTROLLER1
listener.security.protocol.map=CONTROLLER1:PLAINTEXT,CONTROLLER2:PLAINTEXT,CONTROLLER3:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT
Here is the code for the Broker,Controller
process.roles=broker,controller
node.id=2
controller.quorum.voters=1@localhost:9193,2@localhost:9293
listeners=CONTROLLER2://localhost:9293,BROKER2://localhost:9292
controller.listener.names=CONTROLLER2
listener.security.protocol.map=BROKER2:PLAINTEXT,CONTROLLER2:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT
advertised.listeners=BROKER2://localhost:9292
inter.broker.listener.name=BROKER2
And here is the code for the Broker
process.roles=broker
node.id=3
controller.quorum.voters=1@localhost:9193,2@localhost:9293
listeners=BROKER3://localhost:9392
controller.listener.names=CONTROLLER1
listener.security.protocol.map=BROKER3:PLAINTEXT,CONTROLLER1:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
inter.broker.listener.name=BROKER3
advertised.listeners=BROKER3://localhost:9392
The rest of the settings are pretty much default.
I am able to run all 3 of them using the CLI command and they seem to start without any errors
bin/kafka-storage.sh format -t ffQlsx4mQn-ipKduywm2Ig -c config/server.properties --ignore-formatted
bin/kafka-server-start.sh config/server.properties
However, the issue I'm getting is when I try to produce/consume messages to a test topic (I'm running with num.partitions=10)
[2023-09-12 11:32:13,844] WARN [Producer clientId=console-producer] 10 partitions have leader brokers without a matching listener, including [test-4, test-7, test-14, test-9, test-3, test-16, test-0, test-10, test-18, test-13] (org.apache.kafka.clients.NetworkClient)
[2023-09-12 11:32:10,709] WARN [Consumer clientId=console-consumer, groupId=console-consumer-96466] 10 partitions have leader brokers without a matching listener, including [test-4, test-7, test-14, test-9, test-3, test-16, test-0, test-10, test-18, test-13] (org.apache.kafka.clients.NetworkClient)
The CLI commands are
bin/kafka-console-producer.sh --bootstrap-server localhost:9292 --topic test
bin/kafka-console-consumer.sh --bootstrap-server localhost:9292 --topic test --from-beginning
If I try and list existing topics (bin/kafka-topics.sh --list --bootstrap-server localhost:9292), I can see that my test topic was created on both my brokers (9292 and 9392) but the error persists.
Can anybody point me to the right direction? What am I doing wrong?
I'm using:
Your listeners should all have the same names. Don't number them
For instance, inter.broker.listener.name is used for replication, and must match for partitions to be replicated.
Similarly for the controller, the Kraft protocol will become confused in leader election events when controller.listener.names are different between any two nodes in the quorum voters list.