I have the following log file,every Logstash record is supposed to contain multiline lines that end with dash signs '----' ,from the example log you can see that I have four keys, the last key is 'Message' . I have configured the following config file that handles all the lines except the line that starts with 'Message :' As you can see from the log in some of the cases, the 'Message :' contains more than one line, and for these cases, I need that all the lines coming after the 'Message :' will be part of the 'Message :' value and not a separate lines .
Please help me fixing this issue.
log file (input.log)
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Return code : EXCI_NO_ERROR 0
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Abend code : 1564
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Buffer received from xxx
line1
line2
line3
line4
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Return code : EXCI_NO_ERROR 0
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Abend code : 1564
------------------------------------------
config file
input {
file {
path => "/etc/logstash/input.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
# pattern = This says that any line not starting with a '-----' should be merged with the previous line
pattern => "^-----"
negate => true
what => "previous"
}
}
}
filter {
kv {
#source => "message"
field_split => "\n"
value_split => ":"
}
}
output {
file {
path => "/etc/logstash/output.log"
}
#stdout { codec => rubydebug }
}
Take a step back and use other filters type.
You will find below one possible solution.
----\n
from the messageinput {
file {
path => "/tmp/input.log"
start_position => "beginning"
sincedb_path => "/dev/null"
mode => "read"
codec => multiline {
# pattern = This says that any line not starting with a '-----' should be merged with the previous line
pattern => "^-+$"
negate => true
what => "previous"
}
}
}
filter {
mutate { gsub => ["message", "^-+(\n)?", ""] }
if [message] == ""{
drop {}
}
dissect {
mapping => {
"message" => "Timestamp :%{Timestamp}
User :%{User}
Type :%{Type}
Message :%{Message}"
}
}
}
output {
stdout { codec => rubydebug }
}