How send only records that meet some conditions in FluentD

What is a problem?

I want send to ElasticSearch just some record based in conditions.

This is my main conf.

<source>
  bind 0.0.0.0
  <parse>
    expression /<SOME CUSTOM REGEXP>/
    @type regexp
  </parse>
  port 5514
  tag main_tag
  <transport tcp>
  </transport>
  @type syslog
</source>

This is my match for what I send to local3.

<match main_tag.local3.*>
      @type stdout
      @id debug_output
</match>

And a result in /var/log/fluent/fluentd.log

2024-12-11 19:34:57.576728732 +0000 main_tag.local3.info: {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"200","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}

I would like to send just some records to ElasticSearch....

For example just only with "as" value NOT 200.

Or with "x" with value apache OR nginx

Or some other conditions I have to think on.

Or send to different places depends on that conditions.

In short, how manage conditions if is possible.

Environment

- Fluentd version: fluentd 1.17.0 (e763c0761c44d9734b6aa374371387a2e8406522)
- Fluent Package version: fluent-package 5.1.0
- Operating system: Ubuntu 24.04.1 LTS
- Kernel version: 6.8.0-1019-aws

Solution

You can try rewrite_tag_filter plugin with rules configured according to your conditions. First you need to parse syslog source input as json.

Here is an example fluent configuration based on your some of your requirement which I tested with dummy input. Let me know your thoughts!!.

<source>
  @type dummy
  dummy [
    {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"200","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"},
    {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"},
    {"h":"10.0.0.205","x":"nginx","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"},
    {"h":"10.0.0.205","x":"other","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}
  ]
  tag main_tag
</source>

<match main_tag>
  @type rewrite_tag_filter
  <rule>
    key as
    pattern /^200$/
    tag 200.${tag}
  </rule>
   <rule>
    key x
    pattern /^nginx|apache$/
    tag server.${tag}
  </rule>
  <rule>
    key message
    pattern /.+/
    invert true
    tag unmatched.${tag}
  </rule>
  # more rules
</match>

<match 200.main_tag>
  @type stdout
</match>

<match server.main_tag>
  @type stdout
</match>

<match unmatched.main_tag>
  @type stdout
</match>

This should generate following outputs with records matching to respective tags.

2024-12-14 03:35:33 +0000 [info]: #0 starting fluentd worker pid=17 ppid=7 worker=0
2024-12-14 03:35:33 +0000 [info]: #0 fluentd worker is now running worker=0
2024-12-14 03:35:34.079746354 +0000 200.main_tag: {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"200","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}
2024-12-14 03:35:35.082569552 +0000 server.main_tag: {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}
2024-12-14 03:35:36.084676729 +0000 server.main_tag: {"h":"10.0.0.205","x":"nginx","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}
2024-12-14 03:35:37.086780304 +0000 unmatched.main_tag: {"h":"10.0.0.205","x":"other","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}