elasticsearchlogstashkibanaelklogstash-jdbc

Why after set mapping, index return nothing?


I am using Elasticsearch 7.12.0 , Logstash 7.12.0, Kibana 7.12.0 on Windows 10 x64. Logstash config file logistics.conf

input {
  jdbc {
    jdbc_driver_library => "D:\\tools\\postgresql-42.2.16.jar"
    jdbc_driver_class => "org.postgresql.Driver"
    jdbc_connection_string => "jdbc:postgresql://localhost:5433/ld"
    jdbc_user => "xxxx"
    jdbc_password => "sEcrET"
    schedule => "*/5 * * * *"
    statement => "select * from inventory_item_report();"
    }
}

filter {
  uuid {
    target => "uuid"
  }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "localdist"
    document_id => "%{uuid}"
    doc_as_upsert => "true"
    }
}

Run logstash

logstash -f logistics.conf

If I don't set mapping explicit, the query

GET /localdist/_search
{
  "query": {
    "match_all": {}
  }
}

return many result.

My mappings

POST localdist/_mapping
{
  
}

DELETE /localdist

PUT /localdist
{
  
}

POST /localdist
{
  
}

PUT localdist/_mapping
{
  "properties": {
    "unt_cost": {
      "type": "double"
    },
    "ii_typ": {
      "type": "keyword"
    },
    "qty_uom_id": {
      "type": "keyword"
    },
    "prod_id": {
      "type": "keyword"
    },
    "root_cat_id": {
      "type": "keyword"
    },
    "uom": {
      "type": "keyword"
    },
    "product_name": {
      "type": "text"
    },
    "ii_id": {
      "type": "keyword"
    },
    "wght_uom_id": {
      "type": "keyword"
    },
    "iid_seq_id": {
      "type": "long"
    },
    "avai_diff": {
      "type": "double"
    },
    "invt_change_typ": {
      "type": "keyword"
    },
    "ccy": {
      "type": "keyword"
    },
    "exp_date": {
      "type": "date"
    },
    "req_amt": {
      "type": "text"
    },
    "pur_cost": {
      "type": "double"
    },
    "tot_pri": {
      "type": "long"
    },
    "own_pid": {
      "type": "keyword"
    },
    "doc_type": {
      "type": "keyword"
    },
    "ii_date": {
      "type": "date"
    },
    "fac_id": {
      "type": "keyword"
    },
    "shipment_type_id": {
      "type": "keyword"
    },
    "lot_id": {
      "type": "keyword"
    },
    "phy_invt_id": {
      "type": "keyword"
    },
    "facility_name": {
      "type": "text"
    },
    "amt_ohand_diff": {
      "type": "double"
    },
    "reason_id": {
      "type": "keyword"
    },
    "cat_id": {
      "type": "keyword"
    },
    "qty_ohand_diff": {
      "type": "double"
    },
    "@timestamp": {
      "type": "date"
    }
  }
}

run query

GET /localdist/_search
{
  "query": {
    "match_all": {}
  }
}

return nothing.

How to fix it, how to make explicit mappings works correctly?


Solution

  • If I got you right, you are indexing via logstash. Elastic then create the index if missing, indexes the documents, and tries to guess the mapping for your documents based on the very first documents.

    TL;DR: You are DELETING your index containing the data by yourself.

    With

    DELETE /localdist
    

    you are deleting the whole index including all data. After that, by issuing

    PUT /localdist
    {
      
    }
    

    you are re-creating the previously deleted index which is empty again. And at the end, you are setting the index mapping with

    PUT localdist/_mapping
    {
      "properties": {
        "unt_cost": {
          "type": "double"
        },
        "ii_typ": {
          "type": "keyword"
        },
        ...
    

    Now, as you have an empty elastic index with a mapping set, start the logstash pipeline again. If your documents are matching the index mapping, the docs should start to appear very quickly.