kibanalogz.io

How can I export all 8304 hits?

We use logz.io/#/dashboard/kibana to monitor our logs. My filter results in 8304 hits. I want to export them as a CSV for further analysis. When I click on export (next to New, Save, Open, Share, and Auto-refresh) I only get the first 500 hits.

Solution

  • Upd. According to response to a similar question, the increasing the value of discover:sampleSize option will help here: downloaded CSVs should contain more rows.

    However, in his response the member of Elastic team suggests pulling required data by querying Elasticsearch directly should be preferred.

    Try exporting from Elasticsearch directly.

    Consider example from this answer to "Is there any way in Elasticsearch to get results as CSV file in curl api?" question.

    Alternatively, use a tool called elasticsearch-dump. Getting a csv with logs becomes a two steps process:

    1. First, dump contents of desired index/query into a json file:

      elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=query.json \
  --searchBody='{"query":{"term":{"username": "admin"}}}'

    2. Turn json file into csv file (more on this in "Export JSON to CSV with Headers using JQ") using jq command line utility.

    Alternatively, check out this script to dump ES index in csv format.