How do you reconfigure a filter in a Kafka Streams app at runtime (without stopping and restarting the app)? The configuration for what to filter is ideally found in a Kafka configuration topic. The data is actually compacted (KTable) so what I'm trying to achieve is a rewind and reprocessing of all data on filter update as the output topic is also compacted and provides the state of the app after joining a few data tables and applying a filter.
I considered using a join on the configuration topic, but that is problematic because there isn't a primary key or foreign key to join the data topics to. Using flatMap to re-key the configuration topic to match the data topics is problematic because the data topic keys are constantly being generated (new records added). It appears perhaps using an intermediate topic created from the already joined data topic that is then enriched via mapValues could work, though I haven't worked out that API yet either as I need updates to the configuration topic to re-execute the mapValues. This might be something that must be done using the Processor API?
so what I'm trying to achieve is a rewind and reprocessing of all data on filter update as the output topic
This is not possible w/o a restart of the application. You would need to stop the app, and reset the offsets on the input topic, and restart. Kafka Streams does not support seek() so you cannot "rewind" the app while it's running.