IIDR CDC Kafka subscription
For an enterprise wide data streaming platform implementation, we are using a confluent kafka and ibm iidr cdc .
Why the IIDR Management console has only two option to use zookeeper & rest proxy while configuring the subscription for kafka ?
From security perspective , it is not recommended at all to expose zookeeper . How can we just provide them broker endpoints to connect to kafka ? Or how can we implement IIDR confluent kafka connector ? https://www.confluent.io/connector/ibm-data-replication/
How SSL can be implemented in IIDR CDC and Kafka ? any document links ?
Thanks!
Sorry for the delay...
The following document describes several scenarios of IDR Kafka configuration. You can provide broker endpoints editing the /conf/kafkaproducer.properties (on the machine you installed IDR Kafka Agent). If you do so, you still have to provide hostname and port number on MC GUI, but it will ignore them (you can provide fake data in those fields). It also explain some considerations on kerberized environments and on SSH connections.
Keep in mind that JRE provided with CDC has restricted security policy files, so, for using keys greater than 56 bytes (i.e. always), you need to download and upgrade them with unrestricted ones (they can be downloaded from IBM FixCentral page).
I hope this helps.
- Custom kafka connector - timeout on task configuration creation
- Docker - library initialization failed - unable to allocate file descriptor table - out of memory
- What is the ideal number of partitions in kafka topic?
- Unable to consume data using the latest Pyflink Kafka connector
- Kafka Consumer Interceptor not being registered in a spring boot app
- Spring KafkaListener: How to know when it's ready
- Kafka Listener method could not be invoked with the incoming message
- Kafka Msg VS REST Calls
- AWS MSK vs Confluent for hosting Kafka?
- publishEndpoint.Publish() Not Sending Messages with Kafka in MassTransit
- How to use Reactor Kafka KafkaSender API to send messages to two different topics (situated in two different Kafka clusters)?
- Understanding kafka streams partition assignor
- Does delete policy apply to active segments or only to inactive ones?
- Library using sfl4j not following log42.xml configurations
- ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment - Error while installing kafka 3.1.0+java11 in windows
- Kafka producer throws "Received unknown topic or partition error" when sending to topic created via AdminClient createTopics method
- Kafka Connect "Received unknown topic or partition error in fetch for partition..." Overlogging
- Accessing Kafka broker from outside my LAN
- Cannot connect to kafka hosted in cloud from spring boot application
- Kafka producer says "unknown_topic_or_partition"
- dial tcp: lookup ip-x-x-xx.ec2.internal: no such host
- Kafka access inside and outside docker
- How to save multiple partition's Kafka offset by own self when use spark-streaming
- No type attribute provided for component property in log4j2 with Spring Boot
- Confluent Replicator not passing Headers into Converter#fromConnectData
- need to run kafka-storage.sh on every broker and controller
- Kafka topic creation best-practice
- Flink temporal join inconsistencies
- Error Connecting to MSK Cluster with Kafka-Python Using SASL_SSL
- Neo4j CDC Events in Unexpected Format for Property Values