logging systemd ubuntu-server rsyslog systemd-journald

Writing forwarded rsyslogs to journald to be able to filter them by SYSLOG_IDENTIFIER?

Hello StackOverflow Gods/Regulars/Users !

I am currently working on a logging system for two applications running on my servers.

Here is the context :

Server dev1 : Ubuntu server 18.04 (freshly installed)
- is running a systemd.service algo-ep


    [Unit]
    Description="Algo EP"
    [Service]
    Type=simple
    User=me
    WorkingDirectory=/home/me/bin
    ExecStart=/home/me/bin/AlgoEp
    StandardOutput=syslog
    StandardError=syslog
    SyslogIdentifier=dev1_algo_ep
    [Install]
    WantedBy=multi-user.target

is running another systemd.service algo-mdw


    [Unit]
    Description="Algo MDW"
    [Service]
    Type=simple
    User=me
    WorkingDirectory=/home/me/bin
    ExecStart=/home/me/bin/AlgoMdw
    StandardOutput=syslog
    StandardError=syslog
    SyslogIdentifier=dev1_algo_mdw
    [Install]
    WantedBy=multi-user.target

Server dev2 : Ubuntu server 18.04 (freshly installed)
- is running a systemd.service algo-ep


    [Unit]
    Description="Algo EP"
    [Service]
    Type=simple
    User=me
    WorkingDirectory=/home/me/bin
    ExecStart=/home/me/bin/AlgoEp
    StandardOutput=syslog
    StandardError=syslog
    SyslogIdentifier=dev2_algo_ep
    [Install]
    WantedBy=multi-user.target

I wanted to be able to read the log of each service when I ssh on dev1 using journalctl (systemd-journal).

Like this:journalctl -t dev1_algo_ep -t dev1_algo_mdw -t dev2_algo_ep

So, I added a rsyslog.d/algo-ep.conf on dev2:

if $programname == 'dev2_algo_ep' then {          
        action(type="omfwd"                       
           queue.type="linkedlist"                
           queue.filename="algo_fwd"              
           queue.saveOnShutdown="on"              
           action.resumeRetryCount="-1"           
           target="dev1" port="514" protocol="tcp"
        )
}

and added rsyslog.d/algo.conf on dev1:

module(load="imtcp")
module(load="omjournal")

ruleset(name="remote-dev2") {
        action(type="omjournal")
}

input(type="imtcp" port="514" ruleset="remote-dev2")

At this point, no problem, I got the line in journalctl with journalctl -r:

Nov 23 13:27:47 dev1 dev2_algo_ep[3142]:[15246]:  Ep Server listening on localhost:10001...
Nov 23 13:27:47 dev1 dev2_algo_ep[2421]:[15246]:  Ep Server stops...
[...]

But when I try journalctl -t dev2_algo_ep:

me@dev1:~$ journalctl -t dev2_algo_ep
-- Logs begin at Fri 2018-06-01 13:54:11 CEST, end at Fri 2018-11-23 13:27:47 CET. --
me@dev1:~$

Because received log's SYSLOG_IDENTIFIER is set as dev2_algo_ep[3142]: instead of dev2_algo_ep.

So, my question : Is there a way, magical or obvious

to export the log from dev2 to dev1 with a specific SYSLOG_IDENTIFIER ?
or to receive the log on dev1 and to set a specific SYSLOG_IDENTIFIER before sending it to journald ?
or simply to do this ?

Thanks in advance for your advice, your help and your information !

[Edit] It seems that the mix rsyslog + journald is very little known. I didn't found anything in the man page (except the possibility to create a template to rebuild the log at reception on dev1, but looks pretty odd to me).

Solution

option1 since you already got log from journalctl, you can use json format to check what field the line you want, e.g. check what value in SYSLOG_IDENTIFIER field.

the following line shows SYSLOG_IDENTIFIER is nginx, so you can use journalctl -t nginx

journalctl -o json
{ "__CURSOR" : "s=00000000000000000000000000000000;i=146c8;b=170127f0cd5441a3bea76fe326915bd0;m=bb5bba879;t=590367fad5ace;x=33cdab5cd2077705", "__REALTIME_TIMESTAMP" : "1565938628254414", "__MONOTONIC_TIMESTAMP" : "50293614713", "_BOOT_ID" : "170127f0cd5441a3bea76fe326915bd0", "_SELINUX_CONTEXT" : "unconfined\n", "_SYSTEMD_SLICE" : "system.slice", "_MACHINE_ID" : "4211ff3f594041f3966d836585a11a05", "_HOSTNAME" : "VM-0-4-ubuntu", "_TRANSPORT" : "journal", "_CAP_EFFECTIVE" : "0", "PRIORITY" : "5", "_UID" : "102", "_GID" : "106", "_COMM" : "rsyslogd", "_EXE" : "/usr/sbin/rsyslogd", "_CMDLINE" : "/usr/sbin/rsyslogd -n", "_SYSTEMD_CGROUP" : "/system.slice/rsyslog.service", "_SYSTEMD_UNIT" : "rsyslog.service", "_PID" : "3280", "_SYSTEMD_INVOCATION_ID" : "270801fdff7943bc83cd7cef242b3271", "CODE_FILE" : "omjournal.c", "CODE_LINE" : "270", "CODE_FUNC" : "send_non_template_message", "SYSLOG_FACILITY" : "16", "SYSLOG_IDENTIFIER" : "nginx", "MESSAGE" : "36.24.73.111 - - [16/Aug/2019:14:57:08 +0800] \"GET /favicon.ico HTTP/1.1\" 200 946 \"http://111.231.88.233/todo\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36\"", "_SOURCE_REALTIME_TIMESTAMP" : "1565938628254404" }

option2 if you cannot find the SYSLOG_IDENTIFIER, check the rsyslog input module doc. imfile has the tag parameter, so you can use journalctl -t nginx to get the log. check following example

module(load="imfile")
module(load="omjournal")

ruleset(name="nginx") {
    action(type="omjournal")
}

input(
    type="imfile"
    File="/var/log/nginx/access.log"
    tag="nginx"
    ruleset="nginx")

option3

you need check omjournal doc, omjournal has a parameter called tempalte, which you can specify the format you sent to journald check following docs.

www.rsyslog.com/doc/v8-stable/configuration/modules/omjournal.html
www.rsyslog.com/doc/v8-stable/configuration/templates.html
www.freedesktop.org/software/systemd/man/systemd.journal-fields.html

hope it helps.