I am trying to run fluentd as a daemonset on kubernetes cluster (GKE). The config is getting parsed successfully, then the plugins are receiving shutdown signal with few warn messages. There are no error messages. I tried increasing the verbosity level and following is the output of pods:
fluentd-7przp fluentd 2018-09-08 11:02:46 +0000 [info]: #0 fluent/log.rb:322:info: starting fluentd worker pid=9 ppid=1 worker=0
fluentd-7przp fluentd 2018-09-08 11:02:46 +0000 [info]: #0 fluent/log.rb:322:info: fluentd worker is now running worker=0
fluentd-sr764 fluentd 2018-09-08 11:02:50 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.info" time=2018-09-08 11:02:45.151774166 +0000 record={"worker"=>0, "message"=>"fluentd worker is now running worker=0"}
fluentd-dhwnx fluentd 2018-09-08 11:02:51 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.info" time=2018-09-08 11:02:46.029522363 +0000 record={"worker"=>0, "message"=>"fluentd worker is now running worker=0"}
fluentd-7przp fluentd 2018-09-08 11:02:51 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.info" time=2018-09-08 11:02:46.538377182 +0000 record={"worker"=>0, "message"=>"fluentd worker is now running worker=0"}
fluentd-sr764 fluentd 2018-09-08 11:02:55 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.warn" time=2018-09-08 11:02:50.153922217 +0000 record={"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.info", "time"=>2018-09-08 11:02:45.151774166 +0000, "record"=>{"worker"=>0, "message"=>"fluentd worker is now running worker=0"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.info\" time=2018-09-08 11:02:45.151774166 +0000 record={\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}"}
fluentd-sr764 fluentd 2018-09-08 11:03:10 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.warn" time=2018-09-08 11:03:05.168427649 +0000 record={"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.warn", "time"=>2018-09-08 11:03:00.165843014 +0000, "record"=>{"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.warn", "time"=>2018-09-08 11:02:55.156840516 +0000, "record"=>{"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.warn", "time"=>2018-09-08 11:02:50.153922217 +0000, "record"=>{"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.info", "time"=>2018-09-08 11:02:45.151774166 +0000, "record"=>{"worker"=>0, "message"=>"fluentd worker is now running worker=0"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.info\" time=2018-09-08 11:02:45.151774166 +0000 record={\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.warn\" time=2018-09-08 11:02:50.153922217 +0000 record={\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.info\", \"time\"=>2018-09-08 11:02:45.151774166 +0000, \"record\"=>{\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.info\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}\"}"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.warn\" time=2018-09-08 11:02:55.156840516 +0000 record={\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.warn\", \"time\"=>2018-09-08 11:02:50.153922217 +0000, \"record\"=>{\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.info\", \"time\"=>2018-09-08 11:02:45.151774166 +0000, \"record\"=>{\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.info\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.warn\\\" time=2018-09-08 11:02:50.153922217 +0000 record={\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.info\\\", \\\"time\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\"record\\\"=>{\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}, \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.info\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"}\\\"}\"}"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.warn\" time=2018-09-08 11:03:00.165843014 +0000 record={\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.warn\", \"time\"=>2018-09-08 11:02:55.156840516 +0000, \"record\"=>{\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.warn\", \"time\"=>2018-09-08 11:02:50.153922217 +0000, \"record\"=>{\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.info\", \"time\"=>2018-09-08 11:02:45.151774166 +0000, \"record\"=>{\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.info\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.warn\\\" time=2018-09-08 11:02:50.153922217 +0000 record={\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.info\\\", \\\"time\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\"record\\\"=>{\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}, \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.info\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"}\\\"}\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.warn\\\" time=2018-09-08 11:02:55.156840516 +0000 record={\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.warn\\\", \\\"time\\\"=>2018-09-08 11:02:50.153922217 +0000, \\\"record\\\"=>{\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.info\\\", \\\"time\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\"record\\\"=>{\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}, \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.info\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"}\\\"}, \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.warn\\\\\\\" time=2018-09-08 11:02:50.153922217 +0000 record={\\\\\\\"error\\\\\\\"=>\\\\\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\\\\\", \\\\\\\"location\\\\\\\"=>\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\", \\\\\\\"tag\\\\\\\"=>\\\\\\\"fluent.info\\\\\\\", \\\\\\\"time\\\\\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\\\\\"record\\\\\\\"=>{\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"}, \\\\\\\"message\\\\\\\"=>\\\\\\\"dump an error event: error_class=NoMethodError error=\\\\\\\\\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\\\\\\\\\" location=\\\\\\\\\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\\\\\\\\\" tag=\\\\\\\\\\\\\\\"fluent.info\\\\\\\\\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\\\\\\\\\\\\\"worker\\\\\\\\\\\\\\\"=>0, \\\\\\\\\\\\\\\"message\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\"fluentd worker is now running worker=0\\\\\\\\\\\\\\\"}\\\\\\\"}\\\"}\"}"}
fluentd-7przp fluentd 2018-09-08 11:03:24 +0000 [debug]: #0 fluent/log.rb:302:debug: preparing shutdown output plugin type=:elasticsearch_dynamic plugin_id="kubelet_out_es"
fluentd-7przp fluentd 2018-09-08 11:03:24 +0000 [info]: #0 fluent/log.rb:322:info: shutting down output plugin type=:elasticsearch_dynamic plugin_id="kubelet_out_es"
fluentd-dhwnx fluentd 2018-09-08 11:03:25 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.debug" time=2018-09-08 11:03:24.151685730 +0000 record={"message"=>"fluentd main process get SIGTERM"}
fluentd-sr764 fluentd-dhwnx fluentd-7przp fluentd fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on filter plugin type=:parser plugin_id="myapp_filter"
fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on filter plugin type=:parser plugin_id="myapp_filter"
fluentd-dhwnxfluentd-sr764 fluentd 2018-09-08 11:03:25 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0
fluentd 2018-09-08 11:03:25 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0
2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:elasticsearch_dynamic plugin_id="kubelet_out_es"
fluentd-7przp fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:elasticsearch_dynamic plugin_id="apiserver_out_es"
fluentd-7przp fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:elasticsearch_dynamic plugin_id="out_es"
fluentd-7przp fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on filter plugin type=:parser plugin_id="myapp_filter"
fluentd-7przp fluentd 2018-09-08 11:03:26 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0
fluentd.conf:
@include systemd.conf
@include kubernetes.conf
# Start of fluent.conf
<filter kubernetes.var.log.containers.myapp-provider**.log>
@type parser
@id myapp_filter
key_name log
reserve_data true
remove_key_name_field true
<parse>
@type multiline
format_firstline /^[A-Z]/
format1 /^(?<level>[^ ]*)[ \t]+\[(?<time>[^\]]*)\] \[(?<thread>[^\]]*)\] \[(?<request>[^\]]*)\] (?<class>[^ ]*): (?<message>.*)$/
time_format %Y-%m-%d %H:%M:%S,%L %z
</parse>
</filter>
<match kubelet>
@type elasticsearch_dynamic
@id kubelet_out_es
log_level info
include_tag_key true
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
#user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
#password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
index_name fluentd-${tag_parts[0]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}
include_timestamp true
#logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
#logstash_format true
<buffer>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 2M
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
</match>
<match kube-apiserver>
@type elasticsearch_dynamic
@id apiserver_out_es
log_level info
include_tag_key true
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
#user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
#password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
index_name fluentd-${tag_parts[0]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}
include_timestamp true
#logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
#logstash_format true
<buffer>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 2M
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
</match>
<match **>
@type elasticsearch_dynamic
@id out_es
log_level info
include_tag_key true
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
#user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
#password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
index_name fluentd-${record['kubernetes']['container_name']}-${Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}
include_timestamp true
#logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
#logstash_format true
<buffer>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 2M
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
</match>
Can anyone please help to debug this. Thanks in advance!
This could be a kubernetes bug as mentioned here - https://github.com/kubernetes/kubernetes/issues/51376
Although this issue is closed as of now, I would suggest you to try running the same daemonset in default namespace.
Even I tried running it in kube-system namespace but its failing and was running successfully in the default namespace.