flumeflume-ng

Flume replicate then filter, how?


I need to replicate a source, then filter only one of the replicas on a regex.

For example, if my source data was 'dog, donkey, cat' I'd like to sink 'dog, donkey, cat' to one place, but sink only 'dog, donkey' to another place (d*).

I'm planning to use the regex interceptor but I'm not seeing how to do this since the interceptor is associated with the source. How could this be accomplished?


Solution

  • Had to make another flume agent tier for this. Replicated on current agents, and forwarded to new flume agent processes where it could attach the regex interceptor to the source.