I have a Strimzi cluster setup with the follow yaml.
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: kafka
spec:
kafka:
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: tls
- name: external
port: 9094
type: route
authentication:
type: scram-sha-512
tls: true
The pods come up fine and I've created a KafkaUser CR with SCRAM-512 like the following -
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
name: scram-user
labels:
strimzi.io/cluster: kafka
spec:
authentication:
type: scram-sha-512
I've extracted the SCRAM password from the secret properly as well as gotten the ca.crt file from the cluster-ca-cert secret. I'm trying to follow the Go Sarama Code from this sample here - https://github.com/Shopify/sarama/blob/master/examples/sasl_scram_client/main.go
I've also properly gotten the bootstrap server address from the OpenShift Route but I can't seem to connect.
go run sarama.go scram_client.go -brokers bootstrap-address:443 -username scram-user -passwd esoy2WksWRBp -topic test-topic -algorithm sha512 -tls true -ca /path/ca.crt
I've tried a few variations of the above command with adding -certificate or -key flags and none seem to work. Do I have the listener setup wrong?
edit - Forgot to include and mention it but this is what the error I get from the Go Sarama Code.
[Sarama] 2021/08/18 09:22:36 Failed to send SASL handshake kafka-broker:443: x509: certificate signed by unknown authority
[Sarama] 2021/08/18 09:22:36 Closed connection to broker kafka-broker:443
[Sarama] 2021/08/18 09:22:36 client/metadata got error from broker -1 while fetching metadata: x509: certificate signed by unknown authority
[Sarama] 2021/08/18 09:22:36 client/metadata no available broker to send metadata request to
[Sarama] 2021/08/18 09:22:36 client/brokers resurrecting 1 dead seed brokers
[Sarama] 2021/08/18 09:22:36 Closing Client
[Producer] 2021/08/18 09:22:36 failed to create producer: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
exit status 1
So it looks to be an cert issue but I seem to have followed the proper instructions to get the cert. My Kafka broker is just named kafka so the secret is just named kafka-cluster-ca-cert. The ca.crt file is the path I was providing to the Sarama code.
oc get secret kafka-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
Describe of the secret if it matters -
ā°ā oc describe secret kafka-cluster-ca-cert
Name: kafka-cluster-ca-cert
Namespace: strimzi
Labels: app.kubernetes.io/instance=kafka
app.kubernetes.io/managed-by=strimzi-cluster-operator
app.kubernetes.io/name=strimzi
app.kubernetes.io/part-of=strimzi-kafka
strimzi.io/cluster=kafka
strimzi.io/kind=Kafka
strimzi.io/name=strimzi
Annotations: strimzi.io/ca-cert-generation: 0
Type: Opaque
Data
====
ca.crt: 1854 bytes
ca.p12: 1687 bytes
ca.password: 12 bytes
So it turns out that the problem was mainly a command line issue. I kept trying to use the -ca flag when I should have just used only the -certificate flag. I also needed to add the -verify option flag as well. So the command that allowed me to produce was using the following -
go run sarama.go scram_client.go -brokers <your-kafka-boostrap-address>:443 -username <your-scram-username> -passwd <your-scram-password> -topic <your-topic> -algorithm sha512 -tls -certificate <full-path-to-your-cert-file>/ca.crt -verify true
And likewise the command to consume
go run sarama.go scram_client.go -brokers <your-kafka-boostrap-address>:443 -username <your-scram-username> -passwd <your-scram-password> -topic <your-topic> -mode consume -logmsg -algorithm sha512 -tls -certificate <full-path-to-your-cert-file>/ca.crt -verify true
Lesson learned I guess - understand the differences between CA, certificates, and keys.