[SOLVED] Using Rsyslog sending to Loggly and scrub mongodb log data

Using Rsyslog sending to Loggly and scrub mongodb log data

I am trying to send information to loggly via rsyslog with data from mongodb 4.4.2. However I cannot get the data in a way that I can manipulate it and scrub out certain information. When I follow the guide on the Loggly site it works for non mongodb information. If I leave %$!msg% as %msg% I get the mongodb data but I am not able to manipulate it.

config file for reading mongo logs

module(load="imfile")

#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-bundle.crt

# Input for FILE1
input(type="imfile" tag="mongo_lou_qa" ruleset="filelog" file="/var/log/mongodb/mongod.log") #wildcard is allowed at file level only

$template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [token@41058  tag=\"tag1\" tag=\"tag2\" ] %$!msg%" 

set $!msg = $msg;

if re_match($!msg,'([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])')    
then 
{
  set $!ext = re_extract($!msg,'([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])',0,1,"");
  set $!msg= replace($!msg, $!ext, "xxxxxxxxx");
}
  
ruleset(name="filelog"){
action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="LogglyFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
}

Mongodb sample log

{"t":{"$date":"2021-01-01T00:00:00.000-00:00"},"s":"I",  "c":"ACCESS",   "id":20000,   "ctx":"conn79","msg":"Successful authentication","attr":{"mechanism":"ABC","principalName":"__system","authenticationDatabase":"local","client":"0.0.0.0:00000"}}

Solution

rulesets are used to partition the inputs so that only some rules apply to them. It seems that variables like $!msg and $.msg are local to each ruleset, so setting them outside a ruleset will not affect the values in an action that is inside a ruleset. The documentation is not very clear on just what else needs to be inside the ruleset, for example the template probably does not need to be.