elasticsearchsearchshingles

why is shingle token filter with analyser isn't yielding expected results?


Hi here are my index details:

PUT shingle_test
{
  "settings": {
    "analysis": {
      "analyzer": {
        "evolutionAnalyzer": {
          "tokenizer": "standard",
          "filter": [
            "standard",
            "custom_shingle"
          ]
        }
      },
      "filter": {
        "custom_stop": {
            "type": "stop",
            "stopwords": "_english_"
        },
        "custom_shingle": {
            "type": "shingle",
            "min_shingle_size": "2",
            "max_shingle_size": "10",
            "output_unigrams": false
        }
      }
    }
  }, 
  "mappings": {
    "legacy" : {
      "properties": {
        "name": {
          "type": "text",
          "fields": {
            "shingles": {
              "type": "text",
              "analyzer": "standard",
              "search_analyzer": "evolutionAnalyzer"
            },
            "as_is": {
              "type": "keyword"
            }
          },
          "analyzer": "standard"
        }
      }
    }
  }
}

Added 2 docs

PUT shingle_test/legacy/1
{
  "name": "Chandni Chowk 2 Banglore"
}

PUT shingle_test/legacy/2
{
  "name": "Chandni Chowk"
}

Nothing is being returned if I do this,

GET shingle_test/_search
{
  "query": {
    "match": {
      "name": {
        "query": "Chandni Chowk",
        "analyzer": "evolutionAnalyzer"
      }
    }
  }
}

Looked at all possible solutions online, didn't get any.

Also, if I do "output_unigrams": true, then it just works like match query and gives results.

The thing I'm trying to achieve:

Having these documents:

  1. Chandni Chowk 2 Bangalore
  2. Chandni Chowk
  3. CCD Bangalore
  4. Istah shawarma and biryani
  5. Istah

So, searching for "Chandni Chowk 2 Bangalore" should return 1, 2

searching for "Chandni Chowk" should return 1, 2

searching for "Istah shawarma and biryani" should return 4, 5

searching for "Istah" should return 4, 5

searching for "CCD Bangalore" should return 3

note: search keyword will always be exactly equal to value of the name field in the document ex: In this particular index, we can query "Chandni Chowk 2 Bangalore", "Chandni Chowk", "CCD Bangalore", "Istah shawarma and biryani", "Istah". "CCD" won't be queried on this index.


Solution

  • The analyzer parameter specifies the analyzer used for text analysis when indexing or searching a text field.

    Modify your index mapping as

    {
      "settings": {
        "analysis": {
          "analyzer": {
            "evolutionAnalyzer": {
              "tokenizer": "standard",
              "filter": [
                "standard",
                "custom_shingle"
              ]
            }
          },
          "filter": {
            "custom_stop": {
                "type": "stop",
                "stopwords": "_english_"
            },
            "custom_shingle": {
                "type": "shingle",
                "min_shingle_size": "2",
                "max_shingle_size": "10",
                "output_unigrams": true         // note this
            }
          }
        }
      }, 
      "mappings": {
        "legacy" : {
          "properties": {
            "name": {
              "type": "text",
              "fields": {
                "shingles": {
                  "type": "text",
                  "analyzer": "evolutionAnalyzer",         // note this
                  "search_analyzer": "evolutionAnalyzer"
                },
                "as_is": {
                  "type": "keyword"
                }
              },
              "analyzer": "standard"
            }
          }
        }
      }
    }
    

    And, the modified search query will be

    {
      "query": {
        "match": {
          "name.shingles": {
            "query": "Chandni Chowk"
          }
        }
      }
    }
    

    Search Results:

    "hits": [
          {
            "_index": "66127416",
            "_type": "_doc",
            "_id": "2",
            "_score": 0.25759193,
            "_source": {
              "name": "Chandni Chowk"
            }
          },
          {
            "_index": "66127416",
            "_type": "_doc",
            "_id": "1",
            "_score": 0.19363807,
            "_source": {
              "name": "Chandni Chowk 2 Banglore"
            }
          }
        ]