[SOLVED] Logstash Elasticsearch Input Plugin for streaming data

Logstash Elasticsearch Input Plugin for streaming data

I would like to know if we can use logstash-input-elasticsearch plugin for streaming data , ex: if I have data available in my database and I run ElasticSearch input plugin , it will index the data into an output, but if after some time more data comes of ElasticSearch database , Is ElasticSearch input plugin is able to index that data without restarting the logstash ?

Thank you for your attention and your help.

Solution

By default, the elasticsearch input will run a scroll query on your ES instance, pump the documents matching the query, stream them to your outputs and when done Logstash will shut down.

However, what you can do is set up a cron job that will start your Logstash pipeline at regular intervals and queries the new documents present in ES.

UPDATE:

In recent versions of Logstash, the elasticsearch input plugin now provides the ability to schedule the input to run at regular time intervals, much like the jdbc input. There's no need to run Logstash on cron anymore.