elasticsearchsearchkickopensearch

Text fields are not optimised for operations that require per-document field data


Having switched from Elasticsearch to Opensearch, my application now fails to run a simple query with:

"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [status] in order to load field data by uninverting the inverted index. Note that this can use significant memory."

There's a question concerning the same error at Searchkick / Elasticsearch Error: Please use a keyword field instead. Alternatively, set fielddata=true on [name], but there the problem was only affecting tests and I only get the problem (so far) in development mode.

Here's the query being run:

::Record.search(q ? q : "*",
                where: where_clause,
                fields: fields,
                match: :word_middle,
                per_page: max_per_page(per_page) || 30,
                page: page || 1,
                order: sort_clause,
                aggs: aggs,
                misspellings: {below: 5}

If I take out aggs then the search is fine, but they're essential for the application. Removing :status from the list of aggregation fields causes the error to name the next field in the array as the problem. So, I presumably need to specify the correct type for each field used in aggregations. But how?

The Searchkick docs suggest this example under "Advanced Mapping" (https://github.com/ankane/searchkick):

class Product < ApplicationRecord
  searchkick mappings: {
    properties: {
      name: {type: "keyword"}
    }
  }
end

So, I tried this:

# in models/Record.rb
mapping_properties = {}
aggregation_fields.each do |af|
  mapping_properties[af] = { type: 'keyword' }
end
searchkick mappings: {
             properties: mapping_properties
          }

But, the same problem persists. I also tried something similar to that shown in the linked post, e.g.

mappings: {
      properties: {
        name: {
          type: "text",
          fielddata: true,
          fields: {
            keyword: {
              type: "keyword"
            }
          }
        }
      }
    }

...but similarly without luck.

Can anyone suggest how this might be fixed?


Solution

  • This was not related to using Opensearch. Instead, the method of indexing used caused the indexes to be generated incorrectly.

    See: Searchkick memory leak