Can an ElasticSearch ingest pipeline use a search template as its script?
Specifically, I'd like to configure an ingest pipeline so that whenever data of a particular type come in, we query ElasticSearch for some related data, and populate a field (or fields) on the original data based on the results, before loading it in.
I see that ingest pipelines can use scripts (https://www.elastic.co/guide/en/elasticsearch/reference/master/script-processor.html), and that scripts can include search templates (https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html), but I haven't been able to find any information on potentially combining the two.
Actually, the feature you're asking for is coming in 7.5 via the new enrich processor, which kind of provides index-time JOIN capability.
The main idea is to set up an enrich policy that will source data from your related indexes into a new "enrich index" and then you can leverage that "enrich index" in your ingest pipeline using an enrich processor in order to enrich your documents with related fields.
So, without going in too many details, here is how it works in practice:
A
with fields (a
, b
, c
, d
) that you'd like to use for enriching your incoming documentsA
and the "join" field a
z
of the incoming document against field A.a
of the enrich indexb
, c
and d
from the index A
. Note that it will also get the match field a that you can remove using a remove
processor if needed.That's pretty much what you expect. You can find a complete example here. At the beginning, it will work for exact matches (i.e. term
query) and geo matches (i.e. geo_shape
query), but they will probably add new kind of matches (like range
matches) in the near future.