elasticsearchkibanaelasticsearch-painlessnested-fields

How to create an array type runtime field based on a nested field in elasticsearch?


I have an elasticsearch index with a nested field 'roles':

"roles": {
    "type": "nested",
    "properties": {
        "name": {
            "type": "text",
            "fields": {
                "raw": {
                    "type": "text",
                    "analyzer": "keylower"
                }
            }
        },
        "responsibilities": {
            "properties": {
                "name": {
                    "type": "text",
                    "fields": {
                        "raw": {
                            "type": "text",
                            "analyzer": "keylower"
                        }
                    }
                }
            }
        }
    }
}

The values in these fields are arrays, for eg.:

"roles": [
        {
            "name": "System Analyst",
            "responsibilities": [
                {
                    "name": "Software Development"
                },
                {
                    "name": "Software Testing"
                }
            ]
        },
        {
            "name": "Data Analyst",
            "responsibilities": [
                {
                    "name": "Data analysis"
                },
                {
                    "name": "Reporting"
                }
            ]
        }
    ]

I want to build Kibana visualizations on these fields separately. Since it is a nested field and kibana doesn't support it yet (?), I thought of creating runtime fields for each of these fields.

This is the query I used for roles:

PUT employee/_mappings
{
  "runtime": {
    "empRoles": {
      "type": "keyword",
      "script": """if (doc["roles.name.raw"].size()!=0 ) {
        String[] empRoles;
        for(int i=0; i < doc["roles.name.raw"].size(); i++) {
          empRoles[i] = doc["roles.name.raw"].value ;
          
        }
         emit(empRoles);}"""
    }
  }
}

But I am getting the following error:

"caused_by" : {
        "type" : "class_cast_exception",
        "reason" : "Cannot cast from [java.lang.String[]] to [java.lang.String]."
      }

How can I make it work? Ultimately, I want to build a kibana dashboard on the fields 'roles' and 'responsibilities'.


Solution

  • It seems there is no way to use nested fields with runtime fields.

    I could get the desired result by defining a scripted field as suggested here.

    Add the following scripts at Index Pattern > [index pattern name] > Scripted fields tab > Add scripted field (Kibana version 7.17. For higher versions, replace Index Patterns with Data Views).

    For roles:

        def names = new String[params._source.roles.length];
        def i= 0;
        for( role in params._source.roles){
          if(role.name != null){
            names[i++] = role.name;
          }
        }
        return names;
    

    For responsibilities:

        def responsibilitiesLength = 0;
        for( role in params._source.roles){
            responsibilitiesLength += role.responsibilities.length;
        }
        def responsibilities = new String[responsibilitiesLength];
        def i= 0;
        for( role in params._source.roles){
            for( responsability in role.responsibilities){
              responsibilities[i++] = responsability.name
            }
        }
        return responsibilities;
    

    (Note: scripted fields can lead to performance issues and hence have been deprecated)