We have a requirement where we need to forward only specific string logs to kibana endpoint/console. Currently we are getting pattern not match line where the matched string not found. How to ignore those logs not to send to forwarder and only send match logs.
<source>
@type tail
path session.txt
pos_file session.txt.pos
tag sessionlog
<parse>
@type regexp
expression ^\<#\>\s+(?<time>\w+/\w+/\w+\s+[:0-9]+)\s+(?<hostname>[-0-9A-Z]+)\s+(?<message>.*Clip.*)$/
</parse>
</source>
<match sessionlog>
@type stdout
</match>
<#> 2019/11/16 13:56:33 ABC-Hostanme 278424 Dispatcher_1 Msg [Unit1] error emitted: '404'from session start: 2021-11-16T08:54:01
<#> 2019/11/16 13:56:33 ABC-Hostanme 278424 Dispatcher_1 Msg [Unit1] clip result: a1=0, a2=217, a3=152475, a4=148692
Result:
[warn]: #0 pattern not match: <#> 2019/11/16 13:56:33 ABC-Hostanme 278424 Dispatcher_1 Msg [Unit1] error emitted: '404'from session start: 2021-11-16T08:54:01
sessionlog: {"hostname":"DESKTOP-3JOOBVV","message":"278424 Dispatcher_1 Msg [Unit1] clip result: a1=0, a2=217, a3=152475, a4=148692"}
We want to get only matched pattern logs.
@sunshine, If the regexp parser cannot extract a match from the log, it will emit that error. So, its recommended that all log lines passing through the regexp parser can be matched by the expression. I recommend you use the grep filter before the regexp parser to avoid those "pattern not match" logs from fluentd.
I've pasted an example below but you can also use <exclude>
blocks in the grep filter. See here for more info and examples: https://docs.fluentd.org/filter/grep
<source>
@type tail
path session.txt
pos_file session.txt.pos
tag sessionlog
</source>
<filter sessionlog>
@type grep
<regexp>
key message
pattern /INCLUDE_PATTERN_HERE/
</regexp>
</filter>
<filter sessionlog>
@type parser
key_name message
reserve_data true
<parse>
@type regexp
expression ^\<#\>\s+(?<time>\w+/\w+/\w+\s+[:0-9]+)\s+(?<hostname>[-0-9A-Z]+)\s+(?<message>.*Clip.*)$/
</parse>
</filter>
<match sessionlog>
@type stdout
</match>