filebeatelk

Enabling dashboards for filebeat


I am trying to develop more visibility around AWS. I'd really like to use the prebuilt dashboards that come with filebeat, but I seem to constantly run into issues with the visualizations for elb and vpcflow logs. My configuration looks like this:

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "localhost:9243"
  protocol: "https"
  username: "kibana_user"
  password: "kibana_password"
setup.dashboards.enabled: true
setup.dashboards.directory: ${path.config}/kibana
setup.ilm.enabled: false
output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "https"
  username: "elastic_user"
  password: "password"
  indices:
  - index: "cloudtrail-%{[agent.version]}-%{+yyyy.MM.dd}"
    when.contains:
      event.dataset: "aws.cloudtrail"
  - index: "elb-%{[agent.version]}-%{+yyyy.MM.dd}"
    when.contains:
      event.dataset: "aws.elb"
  - index: "vpc-%{[agent.version]}-%{+yyyy.MM.dd}"
    when.contains:
      event.dataset: "aws.vpc"
processors:
  - add_fields:
      target: my_env
      fields:
        environment: development

In my dashboards directory I changed the filebeat-* index to vpc-* for Filebeat-aws-vpcflow-overview.json, cloudtrail-* for filebeat-aws-cloudtrail.json and elb-* for Filebeat-aws-elb-overview.json. The cloudtrail dashboard works just fine. I only run into issues with the elb and vpcflow visualizations. None of elb requests visualizations work. The top IP addresses for vpcflow logs do not work either. Here are some screenshots:

enter image description here

enter image description here


Solution

  • For this particular situation if you don't use the deafault filebeaat-* index there are issues getting the prebuilt dashboards to spin up. I dropped the custom indexing that I had in my configuration and I was able to get the dashboards to load properly.