mongodbelasticsearchkibana-5

How do I handle fields in elasticsearch that contain a '_'?

I am using a Mongo-Connector targeting elasticsearch. This works great for keeping elasticsearch up to date, but I have a problem with one of the fields because it contains an '_'. The data is being replicated/streamed from mongodb continually if I run a rename/reindex the new documents will start showing up with underscores again.

Kibana does not support underscores at the start of a field. What is the best practice for handling this?

I have filed an issue with elastic2-doc-manager for Mongo-Connector to support ingest nodes, but this feels like a much bigger issue with kibana all my attempts at fixing this issue using scripted fields and renaming the field have failed.

This seems like a huge problem. I see underscores in data everywhere, seems like a very poor decision on the side of the kibana team.

Kibana Error: enter image description here

I have found some github referencese to this issue, but no work arounds.

Solution

  • Fields beginning with _ are reserved for use within Elasticsearch. Kibana does not support fields with _ currently, at least not yet. A request for this - https://github.com/elastic/kibana/issues/14856 is still open.

    Until then if you would like to use the field in visualizations etc, I believe you need to rename it.

    While you can't rename the field easily without using logstash or filebeat and Mongo-Connector doesn't support either of them you can instead use a scripted field as below to create a new filed and copy the _ field's value. That way you can use the new field to visualize etc. Add a new scripted field for ex. itemType with the below script and see if it works.

    doc['_itemType.keyword'].value

    Please note though that only keyword fields can be used like this, text type fields won't work. If your _itemType field is of type text, modify the mapping to include a sub field keyword of keyword type under _itemType and try the scripted field.