apache-kafkaapache-kafka-connectconfluent-platformconfluent-control-center

Confluent Replicator End to End Latency Metrics Required

Problem Statement: I am trying to figure out end to end latency for replicator from On-prem to AWS data replication. We found in consumer group for replicator we have the option to display the End-to-End latency but it’s not showing any data in the control center as shown in below screenshots. I tried few things as explained below but it’s not working.

What we need to know is:

We tested the consumer group shows the overall messages running behind in the destination cluster. enter image description here

It’s not showing any data in following screen. I tried figuring out why? enter image description here

In Replicator connector I added following property to enable interceptor: enter image description here What confluent says: “To monitor production and consumption in Control Center, installed the Confluent Monitoring Interceptors with your Apache Kafka® applications and configure your applications to use the interceptors on the Kafka messages produced and consumed, that are then sent to Control Center.”

We installed this in Replicator as mentioned below:

{ 
 "name":"replicator",
    "config":{
      ....
     "src.consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
      ....
    }
  }
}

Then I Checked the broker for Confluent Metrics Reporter which is already configured. enter image description here

Thanks :)

Solution

  • Following are the configuration which worked for me.

    FYI..... Control center/Replicator(connect worker) is running on my source cluster.

    consumer.properties

    zookeeper.connect=src-node1:2181,src-node2:2181,src-node3:2181
bootstrap.servers=src-node1:9092,src-node2,src-node3:9092
interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor

    producer.properties

    zookeeper.connect=dst-node1:2181,dst-node2:2181,dst-node3:2181
bootstrap.servers=dst-node1:9092,dst-node2:9092,dst-node3:9092
interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor

    replicator.properties

    #Replication configuration
name=replicator-onprem-to-aws
topic.rename.format=${topic}.replica
replication.factor=1
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
confluent.topic.replication.factor=1

    If you look into above configuration its the bare basic configuration to run the replicator. I am not using anything fancy here.

    then you can run the following command: 

    [root@src-node1 ~]$ /app/confluent-5.3.1/bin/replicator  --cluster.id 1 --consumer.config config/consumer.properties --producer.config config/producer.properties --replication.config config/replicator.properties --whitelist 'test-topic' > replicator.log 2>&1 &