elasticsearchtriggerslogstashincremental-load

How can I do incremental load into Elasticsearch using logstash+jdbc and Database Trigger

I have elasticsearch cluster and i am using logstash + jdbc to load the data.

I Would like to know is there any way that we can do a incremental load based on database trigger instead of depending on query that uses last modified value column

Any help is greatly appreciated

Solution

  • There are some tools in the market depending on the DB you are using.

    Debezium

    Debezium is an open source project that provides a low latency data streaming platform for change data capture (CDC). You setup and configure Debezium to monitor your databases, and then your applications consume events for each row-level change made to the database.

    PGSync

    PGSync is a middleware for syncing data from Postgres to Elasticsearch. It allows you to keep Postgres as your source of truth and expose structured denormalized documents in Elasticsearch.

    ZomboDB

    ZomboDB allows you to use the power and scalability of Elasticsearch directly from Postgres. You don’t have to manage transactions between Postgres and Elasticsearch, asynchronous indexing pipelines, complex reindexing processes, or multiple data-access code paths -- ZomboDB does it all for you.