[SOLVED] Adding custom jmx metrics to google cloud monitoring collectd configuration

Adding custom jmx metrics to google cloud monitoring collectd configuration

I've added the JVM Monitoring plugin as described here

That's all working great and I can, but now I'd like to add more JMX metrics. e.g. MemoryPool specific counters So I've added this config to /opt/stackdriver/collectd/etc/collectd.d/jvm-sun-hotspot.conf

<MBean "jvm_localhost_MemoryPool">
    ObjectName "java.lang:type=MemoryPool,name=*"
    InstanceFrom "name"
    <Value>
        Type "gauge"
        InstancePrefix "memorypool-usage_used"
        Table false
        Attribute "Usage.used"
    </Value>
</MBean>

and

Collect "jvm_localhost_MemoryPool"

in the Connection section

It seems to be valid collectd config but when it's getting sent to Stackdriver/Google Cloud Monitoring it is rejected.

012    {#012      "index": 261,#012      "valueErrors": [#012        {#012          "error": {#012            "code": 3,#012            "message": "Unsupported
 collectd id: plugin: \"jvm\" type: \"gauge\" type_instance: \"memorypool-usagecommitted\""#012          }#012        }#012      ]#012    },#012    {#012
"index": 262,#012      "valueErrors": [#012        {#012          "error": {#012            "code": 3,#012            "message": "Unsupported collectd id: plug
in: \"jvm\" type: \"gauge\" type_instance: \"memorypool-usageinit\""#012          }#012        }#012      ]#012    },#012    {#012      "index": 263,#012
"valueErrors": [#012        {#012          "error": {#012            "code": 3,#012            "message": "Unsupported collectd id: plugin: \"jvm\" type: \"gau
ge\" type_instance: \"memorypool-usagemax\""#012          }#012        }#012      ]#012    },#012    {#012      "index": 264,#012      "valueErrors": [#012
    {#012          "error": {#012            "code": 3,#012            "message": "Unsupported

Now from what I understand it needs to be added as a custom metric, but this document suggests it would be auto created. And indeed when I look at the list of builtin jvm metrics I can't see how they map onto the existing ones in the collectd configuration.

e.g. how is os-open_fd_count mapped to os/open_files ?

It would be helpful to see the actual api request being sent by Google's custom collectd implementation but I can't see a way of increasing logging.

I can see from this post that it's probably custom metrics where I want to be looking but how do I do that in the collectd configuration?

I've tried

InstancePrefix "custom.googleapis.com/memorypool-usage"

but still no joy.

Has anyone done this before or can offer any suggestions on what I'm doing wrong?

Solution

The troubleshooting documents [1] could be helpful to determine what points need to be transformed, as well as to ensure your transformations behave as expected.

[1] https://cloud.google.com/monitoring/agent/custom-metrics-agent#troubleshooting