amazon-web-servicesapache-kafkajmxtrans

Jmxtrans publishing kafka metrics to Cloudwatch


I am retrieving Kafka metric using Jmxtrans via the JMX port, and attempting to publish into CloudWatch. The below is my Jmxtrans configuration:

{
   "servers":[
      {
         "alias":"kafka_broker",
         "host":"localhost",
         "port":"9999",
         "queries":[
            {
               "obj":"kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs",
               "attr":[
                  "99thPercentile",
                  "MeanRate"
               ],
               "resultAlias":"LeaderElectionRateTimeMs",
               "outputWriters":[
                  {
                     "@class":"com.googlecode.jmxtrans.model.output.CloudWatchWriter",
                     "settings":{
                        "namespace":"samplekafkanamespace",
                        "dimensions":[
                           {
                              "name":"InstanceId",
                              "value":"$InstanceId"
                           }
                        ]
                     }
                  }
               ]
            }
         ]
      }
   ]
}

And my /etc/default/jmxtrans file:

http_proxy=http://proxy.com:3128
https_proxy=https://proxy.com:3128
HTTPS_PROXY=https://proxy.com:3128
HTTP_PROXY=http://proxy.com:3128
NO_PROXY=169.254.169.254
no_proxy=169.254.169.254

But the Cloudwatch requests get timed out. Is there a setting I am missing?

2018-10-19 11:25:35 [jmxtrans-result-1] level com.googlecode.jmxtrans.jmx.ResultProcessor [ResultProcessor.java:65] - Could not write results [Result(attributeName=99thPercentile, className=com.yammer.metrics.reporting.JmxReporter$Timer, objDomain=kafka.controller, typeName=type=ControllerStats,name=LeaderElectionRateAndTimeMs, valuePath=[], value=0.0, epoch=1539908694623, keyAlias=LeaderElectionRateTimeMs), Result(attributeName=MeanRate, className=com.yammer.metrics.reporting.JmxReporter$Timer, objDomain=kafka.controller, typeName=type=ControllerStats,name=LeaderElectionRateAndTimeMs, valuePath=[], value=0.0, epoch=1539908694623, keyAlias=LeaderElectionRateTimeMs)] of query Query(objectName=kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs, keys=[], attr=[99thPercentile, MeanRate], typeNames=[], resultAlias=LeaderElectionRateTimeMs, useObjDomainAsKey=false, allowDottedKeys=false, useAllTypeNames=false, outputWriterInstances=[com.googlecode.jmxtrans.model.output.support.ResultTransformerOutputWriter@10934918]) to output writer com.googlecode.jmxtrans.model.output.support.ResultTransformerOutputWriter@10934918
com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to monitoring.ap-southeast-2.amazonaws.com:443 [monitoring.ap-southeast-2.amazonaws.com/103.8.175.93] failed: connect timed out
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
        at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:1320)
        at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:1296)
        at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:1204)
        at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData(AmazonCloudWatchClient.java:1181)
        at com.googlecode.jmxtrans.model.output.CloudWatchWriter$Writer.doWrite(CloudWatchWriter.java:147)
        at com.googlecode.jmxtrans.model.output.support.ResultTransformerOutputWriter.doWrite(ResultTransformerOutputWriter.java:53)
        at com.googlecode.jmxtrans.jmx.ResultProcessor$1.run(ResultProcessor.java:63)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

Is anyone familiar with this tool?


Solution

  • I managed to get the Jmxstans utilising the proxy settings by providing it in the wrapper configuration. I added the parameters as Java Additional Parameter. The wrapper.conf is located in /etc/jmxtrans/wrapper.conf:

    wrapper.java.additional.7=-Dhttp.proxyHost=proxy
    wrapper.java.additional.8=-Dhttp.proxyPort=3128
    wrapper.java.additional.9=-Dhttps.proxyHost=proxy
    wrapper.java.additional.10=-Dhttps.proxyPort=3128