Filebeat 7.12.1 ECK operator 2.2
I'm trying to setup the filbeat for the Nginx-ingress access logs in my ECK stack (installed in GKE). I can access the logs directly in the pod but nothing is coming to my Kibana dashboard. I have set up two filebeat.autodiscover.providers
hints.enabled: true
, which looks for all the containers with co.elastic.logs/enabled: "true"
ingress
. I can confirm that the name of the pod is nginx-ingress-ingress-nginx-controller-xxxx-xxxxx
Below is my Filebeat auto discover content:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: filebeat
namespace: search
spec:
type: filebeat
version: 7.12.1
elasticsearchRef:
name: elastic-search
kibanaRef:
name: kibana-web
config:
filebeat.autodiscover.providers:
- node: ${NODE_NAME}
type: kubernetes
hints.enabled: true
#add_resource_metadata.namespace.enabled: true
hints.default_config.enabled: "false"
- node: ${NODE_NAME}
type: kubernetes
#add_resource_metadata.namespace.enabled: true
hints.default_config.enabled: "false"
templates:
- condition:
contains:
kubernetes.container.name: ingress
config:
- paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
type: container
exclude_lines: ["^\\s+[\\-`('.|_]"]
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
spec:
serviceAccountName: filebeat
automountServiceAccountToken: true
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirstWithHostNet
#hostNetwork: true # Allows to provide richer host metadata
containers:
- name: filebeat
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
requests:
memory: 200Mi
cpu: 0.2
limits:
memory: 300Mi
cpu: 0.4
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
Adding the answer here in case someone else run into this issue.
The issue is how I'm checking the contains condition. It should've been kubernetes.pod.name instead of kubernetes.container.name. So I replaced
- condition:
contains:
kubernetes.container.name: ingress
to
- condition:
contains:
kubernetes.pod.name: ingress
in the above file and things started to work!