elasticsearchlogstashpipelinenull-checklogstash-filter

How can I check nulls in Logstash pipeline in filter plugin?


I have a source which sends data to my Logstash pipeline via Logstash http plugin.

The data model which is sent like:

{
  "myArray": [
    {
      "myGrocery": {
        "myId": "aString",
        "apple": "aString",
        "banana": aNumber,
        "vegetable": {
          "garlic": "aString",
          "onion": "aString",
          "mushroom": "aString",    
        }
      }
    }
  ]
}

My filter plugin in Logstash:

filter {

    split {
        field => "myArray"
        add_field => {
            "my_id" => "%{[myArray][myGrocery][myId]}"
            "apple" => "%{[myArray][myGrocery][apple]}"
            "banana" => "%{[myArray][myGrocery][banana]}"           
            "vegetable" => "%{[myArray][myGrocery][vegetable]}"
        }
        remove_field => "myArray"
    }

    json {
        source => "[vegetable]"
        target => "[vegetable]"
    }

    mutate {
        rename => { "[vegetable][garlic]" => "[vegetable][myNewGarlic]" }
        rename => { "[vegetable][onion]" => "[vegetable][myNewOnion]" }
        rename => { "[vegetable][mushroom]" => "[vegetable][myNewMushroom]" }
    }

    mutate {
        remove_field => [ "host", "headers", "@version", "url", "event", "user_agent", "http" ]
    }

}

Everything is working fine but I want to display "null" if the banana is sent as null and save it as a log message "banana is null" but I do not want to drop the whole information. The result should be like:

"myId": "aString",
"apple": "aString",
"banana": "null",
"myNewGarlic" : "aString",
"myNewOnion": "aString",
"myNewMushroom": "aString",

Is it possible to do this? And how?


Solution

  • I solved it finally. The problem was the place of myArray. I removed it after splitting banana property.

        split {
            field => "myArray"
            add_field => {
                "my_id" => "%{[myArray][myGrocery][myId]}"
                "apple" => "%{[myArray][myGrocery][apple]}"              
                "vegetable" => "%{[myArray][myGrocery][vegetable]}"
            }
        }
    
    
        if [myArray][myGrocery][banana] {
            mutate {           
                add_field => {
                "my_banana" => "%{[myArray][myGrocery][banana]}"
                }
            }
        } else {
            mutate {            
                add_tag => [ "my_banana is null" ]
            }
        }
    
        mutate {       
            remove_field => "myArray"
        }
    

    This added a tags property in ELS showing "my_banana is null" if my_banana null is.

    In these cases, it "banana" returns null

    null
    "false"
    ""
    false
    if there is no field at all