amazon-web-servicesloggingkubernetesfluentdefk

How i filter fluentD logs on kubernetes?


My kubernetes have liveness enable, and it log on application, like this:

kubectl logs -n example-namespace example-app node-app
::ffff:127.0.0.1 - - [17/Sep/2020:14:12:19 +0000] "GET /docs HTTP/1.1" 301 175
::ffff:127.0.0.1 - - [17/Sep/2020:14:13:19 +0000] "GET /docs/ HTTP/1.1" 200 3104
::192.168.0.1    - - [17/Sep/2020:14:13:19 +0000] "GET /home-page HTTP/1.1" 200 3104
::ffff:127.0.0.1 - - [17/Sep/2020:14:13:19 +0000] "GET /docs HTTP/1.1" 301 175
::ffff:127.0.0.1 - - [17/Sep/2020:14:13:22 +0000] "GET /docs/ HTTP/1.1" 200 3104

I Use fluentD to send logs to ClowdWatch. My fluentD configuration:

https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluentd-quickstart.yaml

How can i filter, to fluentD only matches

::192.168.0.1 - - [17/Sep/2020:14:13:19 +0000] "GET /home-page HTTP/1.1" 200 3104

And ignore

::ffff:127.0.0.1 - - [17/Sep/2020:14:13:19 +0000] "GET /docs HTTP/1.1" 301 175

Thanks!


Solution

  • After some research, i found this solution:

    <match kubernetes.var.log.containers.**_kube-system_**>
      @type null
    </match>
    

    and this

      <filter **>
       @type grep
       exclude1 log docs
      </filter>
    

    The reference:

    https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/issues/91

    https://docs.fluentd.org/filter/grep

    EDIT

    or add:

          exclude_path ["/var/log/containers/cloudwatch-agent*", "/var/log/containers/fluentd*", "/var/log/containers/*istio*"]    
    

    this config ignore the source files with pattern istio.