fluentdopensearchopensearch-dashboards

OpenSearch Dashboard time field

I have Fluentd + OpenSearch + OpenSearch Dashboard stack for working with logs. The problem is my time field in Opensearch Dashboard is string, so my filter by time doesn't work.

OpenSearch Dashboard screen

Any body knows what's wrong with my configuration?

Fluentd parser:

<source>
 @type tail
  path /opt/liferay/logs/*.json.log
  pos_file /var/log/td-agent/test1_gpay.pos
  read_from_head true
  follow_inodes true
  refresh_interval 10
  tag gpay1
  <parse>
    @type json
    time_type string
    time_format %Y-%m-%d %H:%M:%S.%L
    time_key time
    keep_time_key true
  </parse>
</source>

My log format is:

{"time":"2023-02-07 14:00:00.039", "level":"DEBUG", "thread":"[liferay/scheduler_dispatch-3]", "logger":"[GeneralListener:82]", "message":"Found 0 tasks for launch."}

And what I have in OpenSearch Dashboard:

screenshot

I tried to use scripted fields in OpenSearch Dashboard, but my filter for time doesn't work.

Solution

  • There is a tag "Scripted fields". enter image description here

    In this tag, you can create a new custom field and convert your string 2023-02-07 14:00:00.039 to a new field with the field_type is Date so that you can use that field to filter.

    For example, I saved date_time as string epoch time. Now I convert to type date below. Then I use that field for filter date time. enter image description here