filebeatelkelastic-beats

If then else not working in FileBeat processor


I'm trying to setup some processors in a filebeat.yml to process some logs before sending to ELK.

An important part of the processing is determining the "level" of the event, which is not always included in the line in the log file.

This is the idea I have for it right now:

# /var/log/messages
- type: log

  processors:
  - dissect:
      tokenizer: "%{month} %{day} %{time} %{hostname} %{service}: {%message}"
      field: "message"
      target_prefix: "dissect"
  - if:
      when:
        regexp:
          message: ((E|e)rror|(f|F)ault)
    then:
      - add_fields:
          target: 'dissect'
          fields:
            level: error
    else:
      - if:
          when:
            regexp:
              message: (W|W)arning
        then:
          - add_fields:
              target: 'dissect'
              fields:
                level: warning 
        else:
          - add_fields:
              target: 'dissect'
              fields:
                level: information 
  - drop_fields: 
      #duplicate
      fields: ["dissect.month","dissect.day","dissect.time","dissect.hostname","message"]


  # Change to true to enable this input configuration.
  enabled: true

  paths:
    - /var/log/messages

I'm still not sure about those patterns I'm trying... but right now I don't think they're what's causing me to fail.

When trying to run filebeat with console output for a test with

filebeat -e -c filebeat.yml

I get the following error:

2022-01-26T17:45:27.174+0200    ERROR   instance/beat.go:877    Exiting: Error while initializing input: failed to make if/then/else processor: missing or invalid condition
Exiting: Error while initializing input: failed to make if/then/else processor: missing or invalid condition

I'm very new to yaml in general, and the only other beat I've done before is an AuditBeat (which works, and has conditions, but not "if"s). Does anyone know what the problem might be?

To clarify: I commented out all other "input" entries, leaving just this one, and still got this error.

Edit: Version: 7.2.0


Solution

  • The if part of the if-then-else processor doesn't use the when label to introduce the condition. The correct usage is:

     - if:
         regexp:
           message: [...]
    

    You have to correct the two if processors in your configuration.

    Additionally, there's a mistake in your dissect expression. {%message} should be %{message}. Also, the regexp for warning should be (W|w)arning not (W|W)arning (both W's are uppercase in your config).

    This is the corrected processors configuration:

    processors:
      - dissect:
          tokenizer: "%{month} %{day} %{time} %{hostname} %{service}: %{message}"
          field: "message"
          target_prefix: "dissect"
      - if:
          regexp:
            message: ((E|e)rror|(f|F)ault)
        then:
          - add_fields:
              target: 'dissect'
              fields:
                level: error
        else:
          - if:
              regexp:
                message: (W|w)arning
            then:
              - add_fields:
                  target: 'dissect'
                  fields:
                    level: warning 
            else:
              - add_fields:
                  target: 'dissect'
                  fields:
                    level: information