We are using Prometheus for some time and really enjoying it.
Few words about What is jmx-exporter
jmx-exporter is a program that reads JMX data from JVM based applications (e.g. Java and Scala) and exposes it via HTTP in a simple text format that Prometheus understand and can scrape.
So let’s get started with my issue …
We configured the kafka with jmx exporter as the following
export KAFKA_OPTS="-javaagent:/home/jmx_prometheus_javaagent-0.11.0.jar=7071:/home/kafka-2_0_0.yml"
this configuration set in ambari under kakfa config
after setting the configuration we restart all 3 kafka brokers
we Check that jmx-exporter HTTP server is listening:
netstat -tlnp | grep 7071
tcp6 0 0 :::7071 :::* LISTEN 63872/java
And scrape the metrics!
curl -s localhost:7071 | grep -i kafka | head
# HELP kafka_log_logcleanermanager_max_dirty_percent Attribute exposed for management (kafka.log<type=LogCleanerManager, name=max-dirty-percent><>Value)
# TYPE kafka_log_logcleanermanager_max_dirty_percent gauge
kafka_log_logcleanermanager_max_dirty_percent 0.0
until now every thing is cool
but when we start to use kafka commands , for example to print the list of topics we get:
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh –zookeeper $zookeeper_server:2181 –list
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.net.httpserver.ServerImpl.bind(ServerImpl.java:133)
at sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54)
at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145)
at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:49)
... 6 more
FATAL ERROR in native method: processing of -javaagent failed
Note – before we add the following line:
export KAFKA_OPTS="-javaagent:/home/jmx_prometheus_javaagent-0.11.0.jar=7071:/home/kafka-2_0_0.yml "
everything was ok
so any advice – how to start to solve this problem from this point ?
more reference -
https://alex.dzyoba.com/blog/jmx-exporter
https://medium.com/@mousavi310/monitor-apache-kafka-using-grafana-and-prometheus-873c7a0005e2
Kafka CLI scripts are making use of the same env vars that the broker is using(in particullar KAFKA_OPTS and JMX_PORT).
That's why they are conflicting.
I was using confluent container(confluentinc/cp-server:5.5.1) with:
JMX_PORT: 9991
KAFKA_OPTS: "-javaagent:/usr/local/bin/jmx_prometheus_javaagent-0.13.0.jar=7071:/etc/jmx-exporter/kafka-2_0_0.yml"
volumes:
- ./jmx_exporter/kafka-2_0_0.yml:/etc/jmx-exporter/kafka-2_0_0.yml
- ./jmx_exporter/jmx_prometheus_javaagent-0.13.0.jar:/usr/local/bin/jmx_prometheus_javaagent-0.13.0.jar
A lot of threads are pointing to JMX_PORT only but that didn't help me. I had to unset both env vars from inside the container before I was able to run CLI commands. It works as follows:
unset JMX_PORT
unset KAFKA_OPTS
kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic your_topic
It will not mangle your broker because those vars are set per session.