So I am working on a project where I need to get a Grafana dashboard up and I found a few but the dashboard (JMX Overview 3457) I am interested in calls the following
jvm_memory_bytes_used{instance=~"$instance"}
The problem is that the metric "jvm_memory_bytes_used" does not exist, NONE of the metrics that start with "jvm_" come back. If they start with "jmx_" or "java_lang" they are returned. I found it on Github at,
https://github.com/prometheus/jmx_exporter
I realize that "jvm_" metrics are calculated but I am not sure where to say "Please provide the 'jvm_' metrics". I cannot use javaagent to start. I can only use "jmx_prometheus_httpserver" because it is an embedded jetty server that I have no control over; other than adding the JMX parameters to the starting at the shell script start options. This Github project allows me to call the jmxUrl and get metrics but the metrics fall short for some reason.
Of course on other projects I use javaagent and all metrics show up. Only when using "jmx_prometheus_httpserver" they don't show up.
I call it like so in standalone mode:
java -Djava.util.logging.config.file=logging.properties -cp collector/target/collector*.jar io.prometheus.jmx.JmxScraper service:jmx:rmi:///jndi/rmi://192.168.150.13:5472/jmxrmi
I call it configured with jmxUrl:
http://192.168.150.13:54502/metrics
but same result no "jvm_" metrics. with a basic config file:
startDelaySeconds: 0
ssl: false
lowercaseOutputName: true
lowercaseOutputLabelNames: true
Any help would be appreciated.
Well, the bad news is that it seems like jvm_ metrics seem to only be available when using javaagent. the project on GitHub did not tell you that. I found the answer on this question. I am linking to it so in case someone comes here first.
https://stackoverflow.com/questions/51372578/no-kafka-metrics-in-grafana-prometheus
the bad part is I don't know what I am going to do since I have an embedded Jetty server whose metrics I need and I don't have control on building the call to get it started. I have looked everywhere using "Embedded Jetty add javaagent" and nothing. If anyone comes across a solution for this please post.