elasticsearchmetricbeat

Change field name with regex in ElasticSearch metricbeat


I'm using the metricbeat system with the graphite module. This is the module configuration:

- module: graphite
  metricsets:
   - server
  protocol: "udp"
  port: 2003
  templates:
   - filter: "*" 
     namespace: "spark_metrics"
     template: "env.job_id.metric*"
     delimiter: "_"

Sometimes I get a metric with a number. The template looks like that:

env.job_id.number.metric*

I have no idea how to extract that. I also think I can't use the 'rename' processor because it seems there is no regex support. As I said the number is just sometimes there. If it is a specific executor metric then the executor number is being added. I want to extract the executor number to a new tag and I don't want it as part of the metric name.

enter image description here

As you can see the number of the executor is part of the field name. I want to extract it to look like this: graphite.spark_metrics.tag.executor_number = 6.

And the metric field look like: graphite.spark_metrics.executor_filesystem_file_read_bytes = 0

Does dose someone have any suggestions?


Solution

  • I used the script processor to solve the problem. The fields that looks like graphite.spark_metrics.10_memoryBlock = 2 will be split into 2 fields:

    graphite.spark_metrics.memoryBlock = 2
    graphite.spark_metrics.tag.executor = 10
    

    And if the field looks like that: graphite.spark_metrics.memoryBlock = 2 nothing will happen.

    function process(event) {
      var metricPrefix = "graphite.spark_metrics";
      var excutorRegex = /^(\d*)_(.*)/;
      var metricObj = event.Get(metricPrefix)
      var keys = Object.keys(metricObj)
    
      for (var i = 0; i < keys.length; i++) {
        var found = keys[i].match(excutorRegex) // "10_memoryBlock" -> ["10_memoryBlock", "10", "memoryBlock"]
        if (found) {
          event.Put(metricPrefix + ".tag.executor", parseInt(found[1]))
          event.Rename(metricPrefix + "." + found[0], metricPrefix + "." + found[2])
          break;
        }
      }
    }