apache-kafkamonitoringscalingtelegrafjmxtrans

Monitoring of dynamic(scale up/down) kafka cluster

We are using kafka cluster and we want monitor this cluster.

Current our approach:

  1. collect jmx metrics (telegraf jolokia plugin or jmxtrans)
  2. store into influx
  3. render via grafana

But we faced with problem of static configuration of kafka broker's list. We can configure only static list via Jmxtrans or Telegraf jolokia plugin.

We would like dynamic list (example configure only kafka cluster zookeeper connection for getting brokers list in each iteration of collecting metrics) for case scaling up/down of kafka.

May be do exist another instruments for monitoring dynamic kafka cluster?

Solution

  • My finally solution next:

    1. Custom bash input plugin (because i don't want build own telegraf with one custom go input plugin and I know not very well go language now:) )
    2. In bash plugin zkCli is used for all kafka nodes discovery
    3. Next bash script post bulk jolokia requet to each kafka node, aggregates and format to influx plugin.