elasticsearchlogstashgrok

elastic logstash multiline + key value when one of the key(Message) is more then one line


I have the following log file,every Logstash record is supposed to contain multiline lines that end with dash signs '----' ,from the example log you can see that I have four keys, the last key is 'Message' . I have configured the following config file that handles all the lines except the line that starts with 'Message :' As you can see from the log in some of the cases, the 'Message :' contains more than one line, and for these cases, I need that all the lines coming after the 'Message :' will be part of the 'Message :' value and not a separate lines .

Please help me fixing this issue.

log file (input.log)

Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Return code : EXCI_NO_ERROR 0
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Abend code : 1564
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Buffer received from xxx
  line1
  line2
  line3
  line4
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Return code : EXCI_NO_ERROR 0
------------------------------------------
Timestamp :2022-11-03 09:42:08.095
User :USER1
Type :warning
Message :Abend code : 1564
------------------------------------------

config file

input {
  file {
         path => "/etc/logstash/input.log"
                 start_position => "beginning"
                 sincedb_path => "/dev/null"

         codec => multiline {
         # pattern = This says that any line not starting with a '-----' should be merged with the previous line
           pattern => "^-----"
           negate => true
           what => "previous"
        }
  }
}

filter {

      kv {
        #source => "message"
        field_split => "\n"
        value_split => ":"
        }

      }


output {

        file {
         path => "/etc/logstash/output.log"
        }

        #stdout { codec => rubydebug }
}


Solution

  • Tldr;

    Take a step back and use other filters type.

    You will find below one possible solution.

    Solution

    1. Mutate, remove the ----\n from the message
    2. Drop empty message
    3. Dissect the message into the necessary fields
    input {
      file {
        path => "/tmp/input.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        mode => "read"
        codec => multiline {
          # pattern = This says that any line not starting with a '-----' should be merged with the previous line
            pattern => "^-+$"
            negate => true
            what => "previous"
        }
      }
    }
    
    filter {
      mutate { gsub => ["message", "^-+(\n)?", ""] }
      if [message] == ""{
        drop {}
      }
      dissect {
        mapping => {
          "message" => "Timestamp :%{Timestamp}
    User :%{User}
    Type :%{Type}
    Message :%{Message}"
        }
      }
    }
    
    output {
      stdout { codec => rubydebug }
    }