At my company the kubernetes cluster is managed by a team, we must provision a namespace and then create our resources. We cannot use features such as hostPath volumes and we cannot create new roles or namespaces, etc.
So looking at an example implementation of the fluentd-elasticsearch container as a DaemonSet, they all appear to be using hostPath volume mounting but I'm not sure why.
For example, I ran through this: https://www.howtoforge.com/create-a-daemonset-in-kubernetes/
And created this:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: my-fluentd-elasticsearch-daemonset
namespace: kube-system
labels:
k8s-app: fluentd-logging
spec:
selector:
matchLabels:
name: fluentd-elasticsearch
template:
metadata:
labels:
name: fluentd-elasticsearch
spec:
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd-elasticsearch
image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
But got this error:
Error creating: pods "fluentd-elasticsearch-" is forbidden: unable to validate against any pod
security policy: [spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed
to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be
used]
So I have a couple of questions:
Is fluentd mounting volumes then reading files in those volumes that get pushed out to elasticsearch?
Docker is storing logs on the node's disk. Fluentd needs to acces this log files somehow; this is why its running as daemonset, you need it to run on every node with hostpath to access log files.
Can I just remove the volume mounting or is that essential to it functioning?
No, you can't "just remove" volume mounting (hostpath) because fluentd will loose access to the log files that docker keeps on nodes.
Is fluentd using the kubernetes API at all?
There is no straightforward answer to this question. There are plugins I have found that can access k8s metadata using k8s api but I haven't found any plugin that would use k8s api to pull logs.
Are there any non-daemonset containers which would just use the kubernetes API to get the pods then use log api to forward to a log db?
Some similar to this is describled in k8s documentation: sidecar container with a logging agent
So yes, you could possibly deploy fluentd as a sidecar to gather and forward logs to the db. Check the docs for more details.