I am trying to view HTTP access logs from Kibana but was unable to do so.
Sample HTTP access logs.
- 127.0.0.1 - - [12/Apr/2021:12:18:45.378 +0530] "GET /pizzashack/1.0.0/menu HTTP/1.1" - - "https://localhost:9443/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36"
- 127.0.0.1 - [12/Apr/2021:12:18:45.378 +0530] "- - " 401 - "-" "-"
Sample fluent.conf file
<source>
@type tail
path /logs/http_access_.log
pos_file /logs/http_access_.log.pos
tag httpaccess
format /(?<host>-\s*[^ ]*)(?<user>\s*[^ ]* [^ ]*\s*)\[(?<dateNtime>[^ ]*\s*[^ ]*)\]\s*(?<request>[^ ]*\s*[^ ]*\s*[^ ]*)\s*(?<statuscode>[^ ]*)\s*(?<size>[^ ]*)\s*(?<url>[^ ]*)\s*(?<message>.*)/
</source>
<match httpaccess>
@type copy
<store>
@type elasticsearch
host elasticsearch
port 9200
include_tag_key true
include_timestamp true
tag_key @log_name2
logstash_format true
logstash_prefix access.${tag}
flush_interval 5s
</store>
</match>
As per this site it seems the format is correct. But with the above, I could not see the logs in Kibana. Appreciate if someone can help me to resolve this.
Thanks.
I was able to make it work with below format.
format_firstline /-\s*\d{3}.\d{0,3}.\d{0,2}.\d{1,2}.\d{1,3}/
format1 /^(?<host>-\s*\d{3}.\d{0,3}.\d{0,2}.\d{1,2}.\d{1,3})(?<user>\s*[^ ]* [^ ]*\s*)\[(?<time>[^ ]*\s*[^ ]*)\]\s*(?<request>[^ ]*\s*[^ ]*\s*[^ ]*)\s*(?<statuscode>[^ ]*)\s*(?<size>[^ ]*)\s*(?<address>[^ ]*)\s*(?<agent>.*)/
Thanks.