jsonloggingkibanaelastic-stackgrok

Parsing error "_grokparsefailure" in LogStash

At first I displayed the logs in Kibana from the syslog and it worked fine. I set it up according to the documentation.

Now I've changed the source of the logs, now it retrieves logs from my web application and although Kibana still displays them kind of correctly, now there're the Tags "_grokparsefailure" which means that there's an error in parsing the logs.

The current filter I have:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

How can I find out where the parsing error is? Is there an online service which would help me create a grok pattern for my new logs? Any other advice?

UPDATE: the logs are in json.

Solution

  • In response to OP's:

    Is there an online service which would help me create a grok pattern for my new logs?

    My favorite tool for testing grok patterns is: http://grokconstructor.appspot.com/do/match

    But I know some prefer: https://grokdebug.herokuapp.com/

    Your logs probably aren't parsing properly because you're using the syslog pattern on logs that aren't in the syslog format.

    EDIT: For json log parsing you may want to look at either the json filter or the json codec