[SOLVED] Split rsyslog files

Split rsyslog files

I am trying to find a way to make rsyslog write logs into multiple files simultaneous, that is received from the same host. For example write to two files, and switch every log line.

I have one host that sends insane amounts of data (600gb a day), and i want write this logs to multiple files instead of one.

This is because of Splunk that read this file for indexing purposes, cant utilize multiple pipelines (aka. multiple CPU threads) on the same file, and this causes an bottleneck.

I was thinking about making rsyslog writing a new file every second, but this does not sound optimal.

Any suggestion is much appreciated :)

I have not been able to find any documentation that describes this ability in rsyslog.

Solution

The solution for this is to use part of the subseconds to split the file randomly (using last 2 digits, nano seconds) to 100 files at the same time.

template(name="template_nano_file_format" type="list") {
  property(name="$.path")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  constant(value="-")
  property(name="timegenerated" dateFormat="subseconds" regex.expression="..$" regex.type="ERE")
  constant(value=".log")
}