I am trying to use kafka rest proxy for AWS MSK cluster.
MSK Encryption details:
Within the cluster
TLS encryption: Enabled
Between clients and brokers
TLS encryption: Enabled
Plaintext: Not enabled
I have created topic "TestTopic" on MSK and then I have created another EC2 instance in the same VPC as MSK to work as Rest proxy. Here are details from kafka-rest.properties:
zookeeper.connect=z-3.msk.xxxx.xx.xxxxxx-1.amazonaws.com:2181,z-1.msk.xxxx.xx.xxxxxx-1.amazonaws.com:2181
bootstrap.servers=b-1.msk.xxxx.xx.xxxxxx-1.amazonaws.com:9096,b-2.msk.xxxx.xx.xxxxxx-1.amazonaws.com:9096
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="username" password="password";
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=/tmp/kafka.client.truststore.jks
I have also created rest-jaas.properties file with below content:
KafkaClient {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="username"
password="password";
};
and then set the java.security.auth.login.config using:
export KAFKA_OPTS=-Djava.security.auth.login.config=/home/ec2-user/confluent-6.1.1/rest-jaas.properties
After this I started Kafka rest proxy using:
./kafka-rest-start /home/ec2-user/confluent-6.1.1/etc/kafka-rest/kafka-rest.properties
But when I tried to put an event on the TestTopic by calling service from postman: POST: http://IP_of_ec2instance:8082/topics/TestTopic I am getting 500 error. But in the EC2 instance I can see error:
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:441)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:291)
at io.confluent.kafkarest.ProducerPool.buildNoSchemaProducer(ProducerPool.java:120)
at io.confluent.kafkarest.ProducerPool.buildBinaryProducer(ProducerPool.java:106)
at io.confluent.kafkarest.ProducerPool.<init>(ProducerPool.java:71)
at io.confluent.kafkarest.ProducerPool.<init>(ProducerPool.java:60)
at io.confluent.kafkarest.ProducerPool.<init>(ProducerPool.java:53)
at io.confluent.kafkarest.DefaultKafkaRestContext.getProducerPool(DefaultKafkaRestContext.java:54)
... 64 more
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:141)
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:106)
at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:92)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:139)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:74)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:120)
at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:449)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:430)
... 71 more
I can also see that value of sasl.jaas.config = null in the ProducerConfig values.
Could someone please help me with this. Thanks in advance!
Finally the issue was fixed. I am updating the fix here so that it can be beneficial for someone:
kafka-rest.properties file should have below text:
zookeeper.connect=z-3.msk.xxxx.xx.xxxxxx-1.amazonaws.com:2181,z-1.msk.xxxx.xx.xxxxxx-1.amazonaws.com:2181
bootstrap.servers=b-1.msk.xxxx.xx.xxxxxx-1.amazonaws.com:9096,b-2.msk.xxxx.xx.xxxxxx-1.amazonaws.com:9096
client.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="username" password="username";
client.security.protocol=SASL_SSL
client.sasl.mechanism=SCRAM-SHA-512
Neither there was a need to create file rest-jaas.properties nor export KAFKA_OPTS was needed.
After these changes, I was able to put the messages in the kafka topic using scram authentication.