What kind of setup I need for - Storing/Aggregating/Reporting Log files in the cloud

I have about 1000 sites on multiple windows clusters. The IIS log files(text documents) are available for each site in a set location on the server. The solution what I am looking at should be able to do following things.

1) Push the log files into the cloud. 2) Read those log files and aggregate data like visits, views, hits by url and store it in the cloud for ease of reporting. 3) Access the aggregated data by a third party reporting solution.

First, i am trying to figure out what are my options and what kind of setup I need.

I have a similar use-case as you. I use the tools NXLog -> Logstash -> Elasticsearch -> Kibana (The ELK stack). I have found this solution easy to use and scale. My use is very concerned with scale (>40,000 logs/second) and high availability (no downtime), and this has been a great solution. I'll address your concerns in the order you raised them:

1. We use NXLog to ship the log files.

   • Install NXLog on each Windows machine that you want to collect logs from. https://www.loggly.com/docs/logging-from-windows/
   • Configure NXLog to convert the IIS logs to JSON https://www.loggly.com/docs/iis-web-server-logs/

2. NXLog ships these logs to Logstash.

   • Install and configure Logstash. Logstash will collect all of your logs (you can even send from additional sources if you choose to Logstash) and normalize the data. https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
   • Configure NXLog to point to the Logstash you just set-up.
   • Install and configure Elasticsearch. Elasticsearch is a highly-scalable full-text search engine that allows you to store and analyze large quantities of data. https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html
   • Configure Logstash to point to that Elasticsearch. Here is a blog post with a sample configuration. https://blog.sstorie.com/importing-iis-logs-into-elasticsearch-with-logstash/
   • Install and configure Kibana for visualization. https://www.elastic.co/downloads/kibana
   • Use Kibana to create and visualize aggregations https://www.elastic.co/blog/kibana-aggregation-execution-order-and-you
3. Which third party reporting solution? Could you give more information and maybe I can help with that too?

More resources about using the ELK stack for a logging use-case:

• http://logz.io/learn/complete-guide-elk-stack
• https://www.elastic.co/webinars/introduction-elk-stack

EDIT: Chartio integrates with Elasticsearch. You asked "Where in the process can I aggregate the data and store it so it is easily consumable by reporting application?" Elasticsearch does the data aggregation and storing. You store the data in Elasticsearch using the stack that I detailed above, and then you can integrate a reporting application such as Chartio.