elasticsearchfilebeatelk

Filebeat not sending data to elasticsearch


I've installed elasticsearch 8.5 and Kibana 8.5 in my kubernetes cluster simply applying the official helm file in the elastic repo. Now I'm trying to install filebeat with the following conf:

    filebeat.inputs:
    - type: container
      paths:
      - "/var/log/app.log"
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            in_cluster: true

and

output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      protocol: https
      ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]

Our apps are writing logs in pod container under /var/log/app.log but it seems filebeat does not read the log or not send it to elasticsearch because no index are created in elastic.

How can I solve the problem? What am I doing wrong?

Thanks


Solution

  • This is a working config (Paths etc. needs adaption)

    daemonset:
      extraEnvs:
        - name: "ELASTICSEARCH_USERNAME"
          valueFrom:
            secretKeyRef:
              name: elasticsearch-master-credentials
              key: username
        - name: "ELASTICSEARCH_PASSWORD"
          valueFrom:
            secretKeyRef:
              name: elasticsearch-master-credentials
              key: password
    
    filebeatConfig:
      filebeat.yml: |
        logging.metrics.enabled: false
        filebeat.inputs:
          - type: container
            paths:
              - /var/log/containers/agri-check*.log
            json:
              keys_under_root: true
              overwrite_keys: true
            processors:
              - add_kubernetes_metadata:
                  host: ${NODE_NAME}
                  matchers:
                    - logs_path:
                        logs_path: "/var/log/containers/"
          - type: container
            paths:
              - /var/log/containers/*.log
            exclude_files: ['.*/agri-check.*$']
            processors:
              - add_kubernetes_metadata:
                  host: ${NODE_NAME}
                  matchers:
                    - logs_path:
                        logs_path: "/var/log/containers/"
    
        output.elasticsearch:
          host: '${NODE_NAME}'
          hosts: "https://elasticsearch-master:9200"
          username: '${ELASTICSEARCH_USERNAME}'
          password: '${ELASTICSEARCH_PASSWORD}'
          protocol: https
          ssl.verification_mode: none