elasticsearchkibanafluentdaccess-log

passing HTTP access logs to Kibana with fluentd


I am trying to view HTTP access logs from Kibana but was unable to do so.

Sample HTTP access logs.

- 127.0.0.1 - - [12/Apr/2021:12:18:45.378 +0530] "GET /pizzashack/1.0.0/menu HTTP/1.1" - - "https://localhost:9443/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36"
- 127.0.0.1 - [12/Apr/2021:12:18:45.378 +0530] "- - " 401 - "-" "-"

Sample fluent.conf file

 <source>
     @type tail
     path /logs/http_access_.log
     pos_file /logs/http_access_.log.pos
     tag httpaccess
     format /(?<host>-\s*[^ ]*)(?<user>\s*[^ ]* [^ ]*\s*)\[(?<dateNtime>[^ ]*\s*[^ ]*)\]\s*(?<request>[^ ]*\s*[^ ]*\s*[^ ]*)\s*(?<statuscode>[^ ]*)\s*(?<size>[^ ]*)\s*(?<url>[^ ]*)\s*(?<message>.*)/

 </source>

 <match httpaccess>
     @type copy
     <store>
         @type elasticsearch
         host elasticsearch
         port 9200
         include_tag_key true
         include_timestamp true
         tag_key @log_name2
         logstash_format true
         logstash_prefix access.${tag}
         flush_interval 5s
     </store>
 </match>

As per this site it seems the format is correct. But with the above, I could not see the logs in Kibana. Appreciate if someone can help me to resolve this.

Thanks.


Solution

  • I was able to make it work with below format.

    format_firstline /-\s*\d{3}.\d{0,3}.\d{0,2}.\d{1,2}.\d{1,3}/
    format1 /^(?<host>-\s*\d{3}.\d{0,3}.\d{0,2}.\d{1,2}.\d{1,3})(?<user>\s*[^ ]* [^ ]*\s*)\[(?<time>[^ ]*\s*[^ ]*)\]\s*(?<request>[^ ]*\s*[^ ]*\s*[^ ]*)\s*(?<statuscode>[^ ]*)\s*(?<size>[^ ]*)\s*(?<address>[^ ]*)\s*(?<agent>.*)/
    

    Thanks.