Nested sorting not working as expected ElasticSearch

Requirement

• sort product with custom_field_values matching custom_field_id

Mapping

{
  "mapping": {
    "product": {
      "properties": {
        "user_id": {
          "type": "integer"
        }
        "custom_field_values": {
          "type": "nested",
          "properties": {
            "custom_field_id": {
              "type": "integer"
            },
            "id": {
              "type": "integer"
            },
            "value": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword"
                }
              }
            }
          }
        }
      }
    }
  }
}

Example Data

{
  {
    "_type": "product",
    "_source": {
      "user_id": 1,
      "custom_field_values": [
        { "id": 1, "custom_field_id": 1, "value": "A"},
        { "id": 2, "custom_field_id": 2, "value": "B"},
        { "id": 3, "custom_field_id": 3, "value": "C"},
      ]
    }
  },
  {
    "_type": "product",
    "_source": {
      "user_id": 2,
      "custom_field_values": [
        { "id": 4, "custom_field_id": 1, "value": "Y"},
        { "id": 5, "custom_field_id": 2, "value": "Z"},
      ]
    }
  },
  {
    "_type": "product",
    "_source": {
      "user_id": 3,
      "custom_field_values": [
        { "id": 6, "custom_field_id": 2, "value": "P"},
        { "id": 7, "custom_field_id": 3, "value": "Q"},
      ]
    }
  }
}

EXPECTED

• I should be able to sort whole product filter by custom_field_values.custom_field_id, sorted by custom_field_values.value

Example Query

{
  "size":100,
  "from":0, 
  "query":{
    "bool":{
      "filter":{
        "match":{
          "user_id":1
        }
      }
    }
  },
  "sort":[
    {
      "custom_field_values.value.keyword":{
        "order":"desc",
        "nested":{
          "path":"custom_field_values",
          "filter":{
            "match":{
              "custom_field_values.custom_field_id": 2
            }
          }
        }
      }
    }
  ]
}

UPDATED QUERY

{
  "size":100,
  "from":0, 
  "query":{
    "bool":{
      "filter":{
        "match":{
          "user_id":1
        }
      }
    },
    "nested": { 
      "path": "comments",
      "filter": {
        "custom_field_values.custom_field_id": 2
      }
    }
  },
  "sort":[
    {
      "custom_field_values.value.keyword":{
        "order":"desc",
        "nested":{
          "path":"custom_field_values",
          "filter":{
            "match":{
              "custom_field_values.custom_field_id": 2
            }
          }
        }
      }
    }
  ]
}

result order should be 2nd product, then 3rd product and 1st product. And if I want to loop through the product and print custom_field_values.value I should get Z, P, B.

So, the problem was with case sensitive data. https://www.elastic.co/guide/en/elasticsearch/reference/current/normalizer.html fixed my issue.

"settings": {
    "analysis": {
      "normalizer": {
        "my_normalizer": {
          "type": "custom",
          "char_filter": [],
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  }

Now we can use this normalizer with our keyword field type:

field :field_name, type: 'keyword', normalizer: 'my_normalizer'

Hope this was helpful.