filterfluent-bitfluent-bit-rewrite-tag

How to overwrite a rewrite_tag in fluent-bit


In my scenario, I'm using a fluent-bit (version 1.6.1) filter to identify a particular log pattern and tag (from.solr.out) it.

[FILTER]
    Name          rewrite_tag
    Match         test_tag
    Rule          $msg "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.*$"  from.solr.out false
    Emitter_Name  re_emitted

Then I modify the record to add a field like below

[FILTER]
    Name record_modifier
    Match from.solr.out
    Record aid 53

Now I need to give a new tag (format.logging) to that modified record which I tried with below code snippet and it doesn't work.

[FILTER]
    Name          rewrite_tag
    Match         from.solr.out
    Rule          $from.solr.out "^.*$"  format.logging false
    Emitter_Name  re_emitted_indexing

Solution

  • Figure out how to get the result that I expected.

    ...
    [2020/10/22 12:49:07] [ info] [sp] stream processor started
    [0] format.logging: [1603351147.631883694, {"msg"=>"2020-10-16 09:29:03.968 WARN  ...
    

    I thought the original message that comes with the tag $msg is disappearing after adding the new tag (from.solr.out). Because in my second filter, I'm adding a record to the second tag (from.solr.out) not the initial tag ($msg).

    Then in the last filter, I was trying to create a new tag (format.logging) by using the second tag (from.solr.out). That's where things have gone wrong. Instead of using the second tag, I tried to use the first tag ($msg) here. Then I got all the changes I needed as the output.

    Changes I was expecting were;

    The final change I've done is

    [FILTER]
        Name          rewrite_tag
        Match         from.solr.out
        Rule          $msg "^.*$"  format.logging false
        Emitter_Name  re_emitted_indexing