I have successfully got Filebeat exporting logs to MSK in plaintext mode. When trying to do the same over SSL I’m getting error messages from the Filebeat side
Publish failed with circuit breaker is open
and SSL Handshake errors in the MSK Cloudwatch Logs.
I don’t need two way verification so I assume I just need to pass Filebeat the ACM-PCA from the MSK console however I still receive SSL handshake problems.
Could anyone please provide a sample Filebeat config for outputs to MSK.
You can create a private certifcate via the ACM Console.
Once done you can export the certificate from ACM and use the cert and key in your filebeat output.
Step One
certificate manager > request a certificate > request a private certificate > select your kafka private ca
For the DNS name wildcard your MSK host name, for example
*.my-kafka-cluster.h6q2kl.c4.kafka.eu-west-2.amazonaws.com
Once done request the certificate
Step Two
Export the Private certificate under actions in certificate manager, click through and you will get the cert, cert chain and key. You will only need the cert and key.
Step Three
Add your cert and key to your instance and configure your filebeat output to use the SSL certs. (You may want to remove the password from the key) if not configure the filebeat keystore to avoid plaintext passwords
openssl rsa -in [original.key] -out [new.key]
output.kafka:
hosts: ["*******"]
topic: "*****"
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
ssl.certificate: "/root/certs/kafka-broker.pem"
ssl.key: "/root/certs/kafka.key"
You should start to see your data in your topic after you restart filebeat