elasticsearchlogstashfilebeatelklogstash-filter

Logstash filter to copy some text from the log message


I'm currently loggin some text message like this:

"Some text: some variable text"
"Some other text: some other variable text"

And I need to copy the variable part of the message to a new field, in order to have two new fields like this

"field1" : " some variable text"
"field2" : "some other variable text"

So far only been able to copy the entire text message to a new field with the filebeat processors, and the same with logstash. I've done some research and I saw I mightbe able to use conditionals, and with the mutate filter:

filter {
  if "Some text" in [message] {
    mutate { add_field =>{ "field1" => }}}

But I'vent found how to copy the "some variable text" into the new field. Does someone knows how to do it?


Solution

  • One solution is to use directly grok filter like this :

      filter{
        grok {
            match => { 
              "message" => "^Some text: %{GREEDYDATA:field1}" 
              }
        }
       grok {
            match => { 
              "message" => "^Some other text: %{GREEDYDATA:field2}" 
              }
        }
    }
    

    The advantage is this is more readable and more easy to maintain than if condition.

    You can change the grok to handle all case in one line :

          filter{
            grok {
                match => { 
                  "message" => "^(Some text: %{GREEDYDATA:field1}|Some other text: %{GREEDYDATA:field2})" 
                  }
            }
    }