elasticsearchelastic-stackapache-kafka-connectsink

Elasticsearch field mapping is showing date for version field

I have recently setup a Kafka sink-connector to an Elastic pipeline and I noticed error in conversion of one field version which is definitely a text value. I checked the input format schema and I see it is being sent as text and I did not find any conversion for mapping of field I would have put. Any suggestion if something I should check to see where the field mapping information is getting populated for only this field ? Also, to update the mapping of field for all indices (index-0001, index-0002) is there an option to update the mapping for alias name ?

Solution

  • No, you can't set a mapping for a alias. It seems to me, that you never defined a mapping by yourself but relying on the dybamic mapping feature of elasticseaech. This feature will analyse the first document carrying a new field and the field value. Based on that analysis elastic will try an educated guess and set a mapping for that field in this mapping of the given index.

    While this feature is very handy it's sometimes not what you expect. The best practice here is to ingest few docs and let elastic do the work. After taht we're taking the auto generated mapping and manually tweaking it until it's perfect. This mapping we're storing in a index template which will be auto-appiled to the new index when it's name matches the templates pattern.

    In your case, look at

    GET index-1/_mapping

    and copy it into a editor in order to fix/complete the mapping.

    Then add the bespoke mapping to a index template like that:

    PUT _index_template/template_1
{
  "index_patterns": ["index-*"],
  "template": {
    "mappings": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "version": {
          "type": "keyword"
        },
       ...more fields...

    The very next index will have a priper mapping and you will be also able to reindex (and delete afterwards) the wrong mapped indicies into new ones using

    POST _reindex
{
  "source" : {"index" : "index-1"},
  "dest" : { "index" : "index-1_new"}
}

    Here are the related docs: