I managed to configure my Syslog-ng v3.38 to receive logs on port 514 and forward them to a remote Kafka topic. I am also adding some custom tags to the messages before sending. So far so good.
The default macros such as HOST, FACILITY and others are sent using uppercase characters. My goal is to use lowercase characters for the keys. I don't care about the value fields. So instead of "HOST", I would like to send "host". I would like to do this for all of the keys. Following is my current kafka.conf located under the conf.d folder:
` kafka.conf: | @define kafka-implementation kafka-c
options {
keep-hostname(yes);
};
source s_net_kafka {
tcp(ip(0.0.0.0) port(514));
udp(ip(0.0.0.0) port(514) so_rcvbuf(26214400));
};
######################################################################################
destination d_kafka {
kafka(
bootstrap-servers("broker:9092")
topic("syslog")
message("$(format-json --scope rfc5424 --scope nv-pairs --pair ts=datetime($ISODATE) client=$(env clientid))")
);
};
######################################################################################
log {
source(s_net_kafka);
destination(d_kafka);
};`
Is it possible to achieve what I am trying to do? I tried a few things with rewrites and templates but I can't seem to get it right. I tried to add "--key lowercase --value lowercase" inside the message field but that did not help.
Thanks
I tried to add "--key lowercase --value lowercase" inside the message field but that did not help:
message("$(format-json --scope rfc5424 --key lowercase --value lowercase --scope nv-pairs --pair ts=datetime($ISODATE) client=$(env clientid))")
I also tried the following:
message("$(format-json --scope selected-macros --key lowercase --value lowercase --exclude-unnamed --exclude-pairs .=_. --escape-special-chars --escape-newlines --escape-solidus --quotes-always --single-line --root .)\n")
I am pasting the answer I received from the syslog-ng project here and closing it:
At the moment, there's no such thing. $(format-json) uses value-pairs and it does have the notion of transforms, but at the moment there are only a limited set of transformations:
add-prefix() -- adds a prefix in front of the key
shift() -- shifts characters left (e.g. removes a number of characters from the front)
repace-prefix() -- change the prefix to something else
shift-levels() -- shift N number of dots leftwards (e.g. shift off entire levels from the key)
I have now opened a pull request to add upper/lower transformations. I recognize this might be too late for you two weeks later, but still maybe it's still interesting.
https://github.com/syslog-ng/syslog-ng/pull/4452