I'm using the metricbeat system with the graphite module. This is the module configuration:
- module: graphite
metricsets:
- server
protocol: "udp"
port: 2003
templates:
- filter: "*"
namespace: "spark_metrics"
template: "env.job_id.metric*"
delimiter: "_"
Sometimes I get a metric with a number. The template looks like that:
env.job_id.number.metric*
I have no idea how to extract that. I also think I can't use the 'rename' processor because it seems there is no regex support. As I said the number is just sometimes there. If it is a specific executor metric then the executor number is being added. I want to extract the executor number to a new tag and I don't want it as part of the metric name.
As you can see the number of the executor is part of the field name. I want to extract it to look like this: graphite.spark_metrics.tag.executor_number = 6
.
And the metric field look like: graphite.spark_metrics.executor_filesystem_file_read_bytes = 0
Does dose someone have any suggestions?
I used the script processor to solve the problem.
The fields that looks like graphite.spark_metrics.10_memoryBlock = 2
will be split into 2 fields:
graphite.spark_metrics.memoryBlock = 2
graphite.spark_metrics.tag.executor = 10
And if the field looks like that: graphite.spark_metrics.memoryBlock = 2
nothing will happen.
function process(event) {
var metricPrefix = "graphite.spark_metrics";
var excutorRegex = /^(\d*)_(.*)/;
var metricObj = event.Get(metricPrefix)
var keys = Object.keys(metricObj)
for (var i = 0; i < keys.length; i++) {
var found = keys[i].match(excutorRegex) // "10_memoryBlock" -> ["10_memoryBlock", "10", "memoryBlock"]
if (found) {
event.Put(metricPrefix + ".tag.executor", parseInt(found[1]))
event.Rename(metricPrefix + "." + found[0], metricPrefix + "." + found[2])
break;
}
}
}