elasticsearchlogstashlogstash-configurationopensearch

Logstash-OpenSearch: index is not created from provided template file

Due to AWS migration I have to migrate from ElasticSearch to OpenSearch. I am using Logstash to populate index with the DB data. In the existing Logstash's elasticsearch output plugin configuration the index was created during the Logstash’s startup by setting manage_template, template_name and template properties. I want to keep the same behaviour for OpenSearch as well. I switched to logstash output opensearch plugin and have a following config:

output {
    opensearch {
      hosts => "http://localhost:9200"
      index => "ps_local_20220810083001"
      document_id => "%{id0}"
      template_name => "cas-template"
      template => "/usr/share/logstash/pipeline/template/cas-template.json"
      manage_template => "true"
      template_overwrite => "true"
    }
}

The Logstash successfully starts,

[2022-08-18T12:09:02,990][INFO ][logstash.outputs.opensearch][programs] Using mapping template from {:path=>"/usr/share/logstash/pipeline/template/cas-template.json"}
[2022-08-18T12:09:02,993][INFO ][logstash.outputs.opensearch][programRules] Using mapping template from {:path=>"/usr/share/logstash/pipeline/template/cas-template.json"}
[2022-08-18T12:09:03,093][INFO ][logstash.outputs.opensearch][programRules] Installing OpenSearch template {:name=>"cas-template"}
[2022-08-18T12:09:03,094][INFO ][logstash.outputs.opensearch][programs] Installing OpenSearch template {:name=>"cas-template"}

but I don't see the index template has been created in OpenSearch. The following API's requests provide confusing results. As I understand, the template has not been created: enter image description here enter image description here

As a result the index documents created using auto-mapping do not correspond the required structure. I tried to create a template manually before starting the Logstash. In that case documents looks good. But I do want the Logstash to create the template and index.

Question: Can anyone tell me how to solve this?

UPDATED: As per this answer, I should be using the old API to check the specific template details GET http://localhost:9200/_template/cas-template. It shows that the template was created but the mapping section is empty: enter image description here

Solution

  • The first screenshot you get from _cat/templates shows that the template exists.

    The template is still a legacy template stored via the _template API and not the new _index_template API.

    So if you run the following, you should find your template:

    GET http://localhost:9200/_template/cas-template

    Now as to why the index is created with a default automatic template and not the one that was just installed, it's because you're using the new composable template format with the old legacy template endpoint. Make sure that your /usr/share/logstash/pipeline/template/cas-template.json file contains the legacy format and not the composable one. The notable difference is that in the new format everything is wrapped in a section called template: {}