elasticsearchelasticsearch-indices

Is there a way on Elasticsearch ingest-pipelines to use convert processor on multiple fields based on field name suffix?


We have an index with default pipeline who converts a field to boolean, like documented here https://www.elastic.co/guide/en/elasticsearch/reference/7.10/convert-processor.html We're using Elasticsearch v7.10.

Is there a way to create a pipeline to convert multiple fields based on field name suffix like '_b'? Something like:

PUT _ingest/pipeline/string-to-bool
{
  "description": "converts fields ending in '_b' to boolean",
  "processors" : [
    {
      "convert" : {
        "field": "*_b",
        "type": "boolean"
      }
    }
  ]
}

Solution

  • Tldr;

    As far as I know the convert processor does not allow fuzzy matching.

    To workaround

    As you proposed you could write a script doing the job.

    PUT _ingest/pipeline/string-to-bool-script
    {
      "processors" : [
        {
            "script": {
              "lang": "painless",
              "source": """
              for (key in ctx.keySet()) {
                if (key.endsWith("_b")) {
                    ctx[key] = Boolean.parseBoolean(ctx[key]);
                }
              }
              """
            }
          }
      ]
    }
    
    POST /_ingest/pipeline/string-to-bool-script/_simulate
    {
      "docs": [
        {
          "_index": "index",
          "_id": "id",
          "_source": {
            "plop_b": "true",
            "_b": "false"
          }
        }
      ]
    }
    

    Keep in mind this is very simplistic, and would only work for strings with value of true / false (not case sensitive).