phpelasticsearchkibanafoselasticabundle

How to use ingest pipelines on already indexed documents?


I've been using FOSElasticaBundle to index my documents (which are entities from a Symfony project kept in a db through Doctrine) into Elastic Search. FOSElastica does an automatic mapping and index after that all the documents.

The problem is that there's some actions I would want to apply on every documents (on those already indexed and those which will be after), so pipelines and painless seems to be a good solution.

But, I can't get to understand how to apply a pipeline to documents that are already indexed, do you have an idea how ?

I've seen that you can add a "pipeline=my_pipeline_name" after an ES request but you can do it for a single document while I'd want it to affect all the documents.


Solution

  • You can use Pipeline while you move your data from one index to another index.

    You would need to make use of Reindex API in order for it to be executed on the data during its movement/ingestion_process from one index to another.

    Note: This is index level operation meaning it would affect all the documents.

    Below is summary of steps:

    Below is how you make use of Reindex API with pipeline

    POST _reindex
    {
      "source": {
        "index": "source_index_name"
      },
      "dest": {
        "index": "temporary_index",
        "pipeline": "some_ingest_pipeline"
      }
    }
    

    Let me know if this helps!