I'm using AWS Fargate and log_router containers with a custom Fluent Bit configuration to route logs. Here is my current configuration:
[FILTER]
Name grep
Match cw
Exclude log unwatedtext
[FILTER]
Name modify
Match cw
Remove source
Remove container_id
Remove container_name
Remove ec2_instance_id
Remove ecs_cluster
Remove ecs_task_arn
Remove ecs_task_definition
[OUTPUT]
Name cloudwatch_logs
Match cw
log_key log
region eu-west-1
log_group_name /ecs/container
log_stream_prefix fluent-bit-
auto_create_group On
log_format json/emf
[OUTPUT]
Name s3
bucket mybucket
total_file_size 100M
log_key log
match *
s3_key_format /$TAG/%Y/%m/%d/
use_put_object on
upload_timeout 1M
region eu-west-1
Currently, the unwatedtext is being removed for both CloudWatch and S3 outputs. However, I want to remove it only for CloudWatch and send all logs, including those with unwatedtext, to S3.
How can I modify my Fluent Bit configuration to achieve this?
Additional Information:
Update - 22-Jul-24
I have found the issue. Any help will be appreciated Match is only working for * and I dont see why this is happening. Here are my updated config.
lua script
function process_record(tag, timestamp, record)
print("Input tag: ", tag)
print("Record: ", record["tag"])
local s3_log = {}
local cw_log = {}
for k, v in pairs(record) do
cw_log[k] = v
cw_log["tag"] = "cw"
s3_log[k] = v
s3_log["tag"] = "s3"
end
print("S3 tag: ", s3_log["tag"])
print("CW tag: ", cw_log["tag"])
print("S3 Log: ", s3_log["log"])
print("CW Log: ", cw_log["log"])
return 2, timestamp, {s3_log, cw_log}
end
config
[SERVICE]
flush 1
daemon off
log_level debug
[FILTER]
Name lua
Match *
script /append_tag.lua
call process_record
[FILTER]
Name lua
Match $tag cw
script /print.lua
call process_record
[FILTER]
Name rewrite_tag
Match cw
Rule $tag .* cw true
Emitter_Name cw_emitted
[OUTPUT]
Name cloudwatch_logs
Match *cw*
# Match *
# Match_regex ^(cw)$
log_key log
region af-south-1
log_group_name /ecs/log
log_stream_prefix fluent-bit-
auto_create_group On
log_format json/emf
After working on this I have finally found the solution. Here is my working config that is pushing data to cw and s3 and removing some logs for cw
logDestinations.conf
[FILTER]
Name lua
Match *
script /script.lua
call process_record
[FILTER]
Name rewrite_tag
Match_regex .*demo.*
Rule $tag "^(cw)$" cw.$container_id true
Emitter_Name re_emitted_cw
[FILTER]
Name grep
Match_regex .*cw.*
Exclude tag s3
[FILTER]
Name grep
Match_regex .*cw.*
Exclude log /.*message\":\"remove.*/
[FILTER]
Name grep
Match_regex .*demo.*
Exclude tag cw
[OUTPUT]
Name cloudwatch_logs
Match_regex .*cw.*
log_key log
region af-south-1
log_group_name /ecs/demo
log_stream_prefix fluent-bit-
auto_create_group On
log_format json/emf
[OUTPUT]
bucket s3_bucket
total_file_size 200M
log_key log
Match_regex .*demo.*
s3_key_format /demo/%Y-%m-%d/$TAG/
use_put_object on
upload_timeout 1M
region af-south-1
Name s3
script.lua
function process_record(tag, timestamp, record)
local s3_log = {}
local cw_log = {}
if record["log"] == nil then
record["log"] = record["message"] or "default_log_message"
end
for k, v in pairs(record) do
cw_log[k] = v
cw_log["tag"] = "cw"
s3_log[k] = v
s3_log["tag"] = "s3"
end
return 2, timestamp, {s3_log, cw_log}
end
Dockerfile
FROM cr.fluentbit.io/fluent/fluent-bit
ADD logDestinations.conf /logDestinations.conf
ADD script.lua /script.lua