elasticsearchkibana

Date difference scripted field in Kibana


I wanted to find the difference between two fields using scripted fields. Here are the two date fields and their format:

start_time - June 17th 2018, 09:44:46.000
end_time - June 17th 2018, 09:44:49.000

Which will give proc_time.

Here's what I am trying to do in scripted fields:

doc['start_time'].date.millis - doc['end_time'].date.millis

But this is returning the processing time which is deducted from epoch time. For example, if my processing time is 2 seconds, then the output will be epoch time - 2 seconds, which is not what I want.

This is the sample doc:

17 Jun 2018 04:14:46    INFO    CSG event file generation started  at: Sun Jun 17 04:14:46 CDT 2018
17 Jun 2018 04:14:46    INFO      Executing CSG file generation process
Warning: Using a password on the command line interface can be insecure.
17 Jun 2018 04:15:57    INFO    Finished at: Sun Jun 17 04:15:57 CDT 2018

Any help would be appreciated.

Update I've got this working with the following painless script:

((doc['csg_proc_end_time'].date.year) * 31536000 + doc['csg_proc_end_time'].date.monthOfYear * 86400 + doc['csg_proc_end_time'].date.dayOfMonth * 3600 + doc['csg_proc_end_time'].date.secondOfDay) - ((doc['csg_proc_start_time'].date.year) * 31536000 + doc['csg_proc_start_time'].date.monthOfYear * 86400 + doc['csg_proc_start_time'].date.dayOfMonth * 3600 + doc['csg_proc_start_time'].date.secondOfDay)

However, I would welcome any other script which does this in a simpler way.

JSON format for added fields:

"fields": {
    "@timestamp": [
      "2018-06-20T04:45:00.258Z"
    ],
    "zimbra_proc_time": [
      0
    ],
    "csg_proc_time": [
      71
    ],
    "perftech_proc_time": [
      0
    ],
    "csg_proc_end_time": [
      "2018-06-17T04:15:57.000Z"
    ],
    "csg_proc_start_time": [
      "2018-06-17T04:14:46.000Z"
    ]
  },

Solution

  • This is what I've done to reproduce your issue and it works properly:

    PUT test/doc/1
    {
      "csg_proc_end_time": "2018-06-17T04:15:57.000Z",
      "csg_proc_start_time": "2018-06-17T04:14:46.000Z"
    }
    

    Now compute the processing time in a script field:

    GET test/_search
    {
      "script_fields": {
        "proc_time": {
          "script": {
            "source": "(doc.csg_proc_end_time.value.millis - doc.csg_proc_start_time.value.millis) / 1000"
          }
        }
      }
    }
    

    Result: 71 seconds

    {
        "_index": "test",
        "_type": "doc",
        "_id": "1",
        "_score": 1,
        "fields": {
          "proc_time": [
            71
          ]
        }
      }