I've installed elasticsearch 8.5 and Kibana 8.5 in my kubernetes cluster simply applying the official helm file in the elastic repo. Now I'm trying to install filebeat with the following conf:
filebeat.inputs:
- type: container
paths:
- "/var/log/app.log"
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
in_cluster: true
and
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
protocol: https
ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]
Our apps are writing logs in pod container under /var/log/app.log but it seems filebeat does not read the log or not send it to elasticsearch because no index are created in elastic.
How can I solve the problem? What am I doing wrong?
Thanks
This is a working config (Paths etc. needs adaption)
daemonset:
extraEnvs:
- name: "ELASTICSEARCH_USERNAME"
valueFrom:
secretKeyRef:
name: elasticsearch-master-credentials
key: username
- name: "ELASTICSEARCH_PASSWORD"
valueFrom:
secretKeyRef:
name: elasticsearch-master-credentials
key: password
filebeatConfig:
filebeat.yml: |
logging.metrics.enabled: false
filebeat.inputs:
- type: container
paths:
- /var/log/containers/agri-check*.log
json:
keys_under_root: true
overwrite_keys: true
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- type: container
paths:
- /var/log/containers/*.log
exclude_files: ['.*/agri-check.*$']
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
output.elasticsearch:
host: '${NODE_NAME}'
hosts: "https://elasticsearch-master:9200"
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl.verification_mode: none