We have OpenSearch 2.5 and filebeat 7.12.1 that collects logs from EKS cluster (v1.27). We no longer get logs from cluster with kubernetes metadata. We get logs for cloud metadata, but we cannot get our kubernetes metadata objects to load into OpenSearch.
This was working previous to our EKS upgrade from v1.23-->1.27. Have been working on this for some time now with another engineer. Any help is appreciated.
Here is our filebeat config
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
logging.level: debug
logging.selectors: [ "kubernetes" ]
filebeat.config:
inputs:
# Mounted `filebeat-inputs` configmap:
path: ${path.config}/inputs.d/*.yml
# Reload inputs configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
#filebeat.autodiscover:
#providers:
# - type: kubernetes
# node: ${NODE_NAME}
# hints.enabled: true
# hints.default_config:
# type: container
# paths:
# - /var/log/pods/*/*/*.log
# templates:
# - condition:
# contains:
# kubernetes.container.name: "no-json-logging"
# config:
# - type: container
# paths:
# - "/var/log/pods/*-${data.kubernetes.container.id}.log"
# - condition:
# contains:
# kubernetes.container.name: "json-logging"
# config:
# - type: container
# paths:
# - "/var/log/pods/*-${data.kubernetes.container.id}.log"
# json.keys_under_root: true
# json.add_error_key: true
# json.message_key: message
processors:
- add_cloud_metadata:
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT:443}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
protocol: 'https'
ilm.enabled: false
setup.ilm.enabled: false
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: kube-system
labels:
k8s-app: filebeat
data:
kubernetes.yml: |-
- type: container
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
symlinks: true
paths:
- /var/log/pods/*/*/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
in_cluster: true
add_resource_metadata:
default_indexers.enabled: false
default_matchers.enabled: false
indexers:
- container:
matchers:
- fields:
lookup_fields: ["container.id"]
- logs_path:
logs_path: '/var/log/pods/'
resource_type: 'container'
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
automountServiceAccountToken: true
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat-oss:7.12.1
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: ELASTICSEARCH_HOST
value: "XXX"
- name: ELASTICSEARCH_PORT
value: "443" # 443
- name: ELASTICSEARCH_USERNAME
value: "XXX"
- name: ELASTICSEARCH_PASSWORD
value: "XXX"
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: inputs
mountPath: /usr/share/filebeat/inputs.d
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/log/pods
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/log/pods
- name: inputs
configMap:
defaultMode: 0600
name: filebeat-inputs
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
---
Tried several variations of the config, but cannot get kubernetes metadata to load. Even tried autodiscover mode, but when I try it no logs get into OpenSearch at all.
Expecting tags to show in OpenSearch for kubernetes - for example:
kubernetes.container.name kubernetes.labels.app kubernetes.labels.env kubernetes.pod.name
We have custom labels in pods to scope logs by env, and app name.
Again, all of this was working before the EKS upgrade.
There are no errors in filebeat logs to indicate a problem. Logs appear to be loading via harvesters and indexes are created. Looks like it should be working but its not. The only thing I see in logs is "...did not match any of the cached resources", but not sure what this means.
Attaching log output:
2023-10-05T02:01:06.506Z INFO instance/beat.go:660 Home path: \[/usr/share/filebeat\] Config path: \[/usr/share/filebeat\] Data path: \[/usr/share/filebeat/data\] Logs path: \[/usr/share/filebeat/logs\]
2023-10-05T02:01:06.506Z INFO instance/beat.go:668 Beat ID: 72dee488-d083-4145-a5a2-ec77566c0519
2023-10-05T02:01:06.509Z INFO \[add_cloud_metadata\] add_cloud_metadata/add_cloud_metadata.go:105 add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"xxx"},"availability_zone":"us-east-1b","image":{"id":"ami-013895
b64fa9cbcba"},"instance":{"id":"i-0c4320f7a209ffa8d"},"machine":{"type":"t3.medium"},"provider":"aws","region":"us-east-1"}
2023-10-05T02:01:06.510Z INFO \[seccomp\] seccomp/seccomp.go:124 Syscall filter successfully installed
2023-10-05T02:01:06.510Z INFO \[beat\] instance/beat.go:996 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "fi
lebeat", "uuid": "72dee488-d083-4145-a5a2-ec77566c0519"}}}
2023-10-05T02:01:06.511Z INFO \[beat\] instance/beat.go:1005 Build info {"system_info": {"build": {"commit": "651a2ad1225f3d4420a22eba847de385b71f711d", "libbeat": "7.12.1", "time": "2021-04-20T19:58:27.000Z", "version": "7.12.1"}}}
2023-10-05T02:01:06.511Z INFO \[beat\] instance/beat.go:1008 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.15.9"}}}
2023-10-05T02:01:06.515Z INFO \[beat\] instance/beat.go:1012 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-09-29T17:25:38Z","containerized":true,"name":"filebeat-74xql","ip":\["127.0.0.1/8","::1/128","172.19.54.66/32","fe80
::6029:bdff:fea6:c030/64"\],"kernel_version":"5.10.186-179.751.amzn2.x86_64","mac":\["62:29:bd:a6:c0:30"\],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"U
TC","timezone_offset_sec":0,"id":"5ea6a666bf6bfd4b2167796371a02dc5"}}}
2023-10-05T02:01:06.517Z INFO \[beat\] instance/beat.go:1041 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":\["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw",
"sys_chroot","mknod","audit_write","setfcap"\],"effective":\["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"\],"bounding":\["chown","dac_override","fowner","fsetid","kill","
setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"\],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 7, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs"
:true}, "start_time": "2023-10-05T02:01:06.430Z"}}}
2023-10-05T02:01:06.517Z INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.12.1
2023-10-05T02:01:06.517Z INFO eslegclient/connection.go:99 elasticsearch url: https://xxx.us-east-1.es.amazonaws.com:443
2023-10-05T02:01:06.518Z INFO \[publisher\] pipeline/module.go:113 Beat name: filebeat-74xql
2023-10-05T02:01:06.518Z INFO \[monitoring\] log/log.go:117 Starting metrics logging every 30s
2023-10-05T02:01:06.518Z INFO instance/beat.go:468 filebeat start running.
2023-10-05T02:01:06.519Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=222664
2023-10-05T02:01:06.523Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=222868
2023-10-05T02:01:06.523Z INFO \[registrar\] registrar/registrar.go:109 States Loaded from registrar: 10
2023-10-05T02:01:06.523Z INFO \[crawler\] beater/crawler.go:71 Loading Inputs: 0
2023-10-05T02:01:06.524Z INFO log/input.go:157 Configured paths: \[/var/log/pods/\*/\*/\*.log\]
2023-10-05T02:01:06.525Z INFO \[crawler\] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0
2023-10-05T02:01:06.525Z INFO cfgfile/reload.go:164 Config reloader started
2023-10-05T02:01:06.525Z INFO cfgfile/reload.go:224 Loading of config files completed.
2023-10-05T02:01:06.525Z INFO cfgfile/reload.go:164 Config reloader started
2023-10-05T02:01:06.529Z INFO log/input.go:157 Configured paths: \[/var/log/pods/\*/\*/\*.log\]
2023-10-05T02:01:06.529Z INFO cfgfile/reload.go:224 Loading of config files completed.
2023-10-05T02:01:06.531Z INFO log/harvester.go:302 Harvester started for file: /var/log/pods/kube-system_metricbeat-ncnsr_90ac5901-9638-4b09-8fdd-3b1f6ff4f648/metricbeat/0.log
2023-10-05T02:01:06.531Z INFO log/harvester.go:302 Harvester started for file: /var/log/pods/kube-system_kube-proxy-2s26d_1b51389c-42b3-4f98-b67a-ce303b3a2c98/kube-proxy/0.log
2023-10-05T02:01:06.531Z INFO log/harvester.go:302 Harvester started for file: /var/log/pods/kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/filebeat/0.log
2023-10-05T02:01:06.550Z INFO add_kubernetes_metadata/kubernetes.go:71 add_kubernetes_metadata: kubernetes env detected, with version: v1.27.4-eks-2d98532
2023-10-05T02:01:06.550Z DEBUG \[kubernetes\] add_kubernetes_metadata/matchers.go:72 logs_path matcher log path: /var/log/pods/
2023-10-05T02:01:06.550Z DEBUG \[kubernetes\] add_kubernetes_metadata/matchers.go:73 logs_path matcher resource type: container
2023-10-05T02:01:06.550Z INFO \[kubernetes\] kubernetes/util.go:99 kubernetes: Using node ip-172-19-55-203.ec2.internal provided in the config {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:06.550Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:162 Initializing a new Kubernetes watcher using host: ip-172-19-55-203.ec2.internal {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:06.650Z DEBUG \[kubernetes\] kubernetes/watcher.go:184 cache sync done
2023-10-05T02:01:06.751Z DEBUG \[kubernetes\] kubernetes/watcher.go:184 cache sync done
2023-10-05T02:01:06.851Z DEBUG \[kubernetes\] kubernetes/watcher.go:184 cache sync done
2023-10-05T02:01:06.851Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: default/ui-blue-0 {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:06.851Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:296 Created index ff93836067b818d36598827bb5f6355e7e7f0306d19de9fda6bca32f3977ef95 for pod default/ui-blue-0 {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:06.851Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/aws-node-n4zb9 {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:06.851Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:296 Created index 2e0feef7d881d27dbf6542a0d588751e661869cfdbb1cf95335618509bc29b6e for pod kube-system/aws-node-n4zb9 {"libbeat.processor": "add_kubernetes_metadata"
}
2023-10-05T02:01:06.851Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:296 Created index 2da92f12efb668e79a589484fa2aedbfdf0620c921b00f6c8c4b37d61cba7c25 for pod kube-system/aws-node-n4zb9 {"libbeat.processor": "add_kubernetes_metadata"
}
2023-10-05T02:01:06.851Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/ebs-csi-node-wk6nn {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:06.851Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:296 Created index e1391f837355370f7733689cb6193c7e034f6be1e8901f08e08e9cdd4ff4aa49 for pod kube-system/ebs-csi-node-wk6nn {"libbeat.processor": "add_kubernetes_metadata"
}
2023-10-05T02:01:06.851Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:296 Created index d790a2114216777abd6f0902265d390a533f1245bcc9956cf560166a56d7a898 for pod kube-system/ebs-csi-node-wk6nn {"libbeat.processor": "add_kubernetes_metadata"
}
2023-10-05T02:01:06.852Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:296 Created index 8e9494a4b785f98acc40ed3cba7a8db78bcc99b7455ef54bc3b601578372c3e2 for pod kube-system/ebs-csi-node-wk6nn {"libbeat.processor": "add_kubernetes_metadata"
}
2023-10-05T02:01:06.852Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/filebeat-74xql {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:06.852Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/kube-proxy-2s26d {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:06.852Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:296 Created index 2b7110b7e3cc61d5fad18595ff0e962a1771ded45133f279fb108637c63305fa for pod kube-system/kube-proxy-2s26d {"libbeat.processor": "add_kubernetes_metadata"
}
2023-10-05T02:01:06.852Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/metricbeat-ncnsr {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:06.852Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:296 Created index 9a0347946c5e0e28598a8ced3ece865308abe39215783125c9c8581fa8c3a9b1 for pod kube-system/metricbeat-ncnsr {"libbeat.processor": "add_kubernetes_metadata"
}
2023-10-05T02:01:07.179Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:211 Updating kubernetes pod: kube-system/filebeat-74xql {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:07.179Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:296 Created index 2103af45628eb99b74286c17859bbeee3a7749d2566aa9794d5400474f0ec71b for pod kube-system/filebeat-74xql {"libbeat.processor": "add_kubernetes_metadata"
}
2023-10-05T02:01:07.532Z INFO \[publisher_pipeline_output\] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://search-preprod-logging-alwadubacsyc4yts5wjecxdfdq.us-east-1.es.amazonaws.com:443))
2023-10-05T02:01:07.532Z INFO \[publisher\] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-10-05T02:01:07.532Z INFO \[publisher\] pipeline/retry.go:223 done
2023-10-05T02:01:07.533Z DEBUG \[kubernetes\] add_kubernetes_metadata/matchers.go:88 Incoming log.file.path value: /var/log/pods/kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/filebeat/0.log
2023-10-05T02:01:07.533Z DEBUG \[kubernetes\] add_kubernetes_metadata/matchers.go:126 Using container id: kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/
2023-10-05T02:01:07.533Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:252 Using the following index key kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/ {"libbeat.processor": "add_kubernetes_metadata"}
2023-10-05T02:01:07.533Z DEBUG \[kubernetes\] add_kubernetes_metadata/kubernetes.go:255 Index key kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/ did not match any of the cached resources {"libbeat.processor": "add_kubernetes_metadata"
Any help is appreciated. Thank you.
We were able to resolve this by removing the add_kubernetes_metadata processor completely and getting the data needed for sorting by parsing the logfile path with the dissect processor, adding a custom field for environment, and manually populating the kubernetes.* labels that our OpenSearch dashboards were expecting.
Also, once dissected, you can immediately use those fields for other processors.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
# Mounted `filebeat-inputs` configmap:
path: ${path.config}/inputs.d/*.yml
# Reload inputs configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
processors:
- add_cloud_metadata:
- add_host_metadata:
- drop_event:
when:
contains:
log.file.path: "kube-system"
- dissect:
tokenizer: "/var/log/pods/%{namespace}_%{pod.name}_%{}/%{container.name/%{logfile}"
field: "log.file.path"
target_prefix: "kubernetes"
- drop_event:
when:
equals:
kubernetes.container.name: "efs"
- rename:
fields:
- from: "myenv"
to: "kubernetes.labels.env"
ignore_missing: false
fail_on_error: true
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT:443}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
protocol: 'https'
ilm.enabled: false
setup.ilm.enabled: false
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: kube-system
labels:
k8s-app: filebeat
data:
kubernetes.yml: |-
- type: log
fields_under_root: true
fields:
myenv: "dev"
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
paths:
- /var/log/pods/**