I have 3 lines of logs with different structure. I am trying to construct a grok pattern to filter the logs.
[2022-10-04 21:45:27,444: INFO/MainProcess] Events of group {task} enabled by remote
[2022-10-04 21:43:06,521: ERROR/MainProcess] consumer: Cannot connect to redis://10.0.13.57:6379/0: Error 111 connecting to 10.0.13.34:6379. Connection refused..
[2022-10-04 21:45:22 +0000] [3094] [INFO] Listening at: http://0.0.0.0:8793 (3094)
I am expecting:
timestamp: loglevel: message:
The grok pattern I have doesn't match anything:
\[%{TIMESTAMP_ISO8601:timestamp}\]\:%{LOGLEVEL:loglevel}%{WORD: class} %{SPACE}%{GREEDYDATA:logMessage}
You need to have two grok pattern for separate logs.
[2022-10-04 21:45:27,444: INFO/MainProcess] Events of group {task} enabled by remote
[2022-10-04 21:43:06,521: ERROR/MainProcess] consumer: Cannot connect to redis://10.0.13.57:6379/0: Error 111 connecting to 10.0.13.34:6379. Connection refused..
The grok pattern for the above two logs:
%{DATESTAMP:timestamp}\: %{LOGLEVEL:loglevel}\/%{DATA:data}\] %{GREEDYDATA:message}
[2022-10-04 21:45:22 +0000] [3094] [INFO] Listening at: http://0.0.0.0:8793 (3094)
The grok pattern for the above log:
\[%{TIMESTAMP_ISO8601:timestamp} \+%{DATA:data}\] \[%{LOGLEVEL:loglevel}\] %{GREEDYDATA:message}
Also, you can make use of the Drop Filter of the logstash to drop the field data generated [see the output screenshot below] after parsing your logs using the above GROK pattern.