elasticsearchmappingartificial-intelligenceknnembedding

dense_vector becomes a field of type "float"


I try to implement a search engine using elasticsearch. For that i created an index with a mapping that contains dense_vector fields. PUT {{elastic uri}}/{{index}}

{
    "mappings": {
        "properties": {
            ...
            "title_embeddings": {
                "type": "dense_vector",
                "index": true,
                "similarity": "cosine",
                "dims": 1024,
                "index_options": {
                    "type": "hnsw",
                    "ef_construction": 100,
                    "m": 16
                }
            },
            "title": {
                "type": "text"
            },
            ...
        }
    }
}

The problem is that when i retrieve it using the API, the field is of type "float" instead of dense_vector.

GET {{elastic-uri}}/{{index}}/_search

{
    "{{index}}": {
        "mappings": {
            "properties": {
                ...
                "title": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "title_embeddings": {
                    "type": "float"
                }
                ...
            }
        }
    }
}

Also i can't use the KNN search on it since it's not the correct type :

GET {{elastic-uri}}/{{index}}/_search

{
  "knn": {
    "field": "title_embeddings",
    "query_vector": [0.1, 3.2, 2.1],
    "k": 2,
    "num_candidates": 100
  }
}

Error : failed to create query: [knn] queries are only supported on [dense_vector] fields


Solution

  • I was indexing my data with the following code :

    es.index(
                index='{{index}}',
                document=json.dumps(page.__dict__),
                error_trace=True
            )
    

    And with this new code it works :

     es.index(
                index='{{index}}',
                document={
                    "id": page.id,
                    "title": page.title,
                    "title_embeddings": page.title_embeddings,
                    ...
                },
                error_trace=True
            )