I am new to flink and looking for advice on building a realtime event correlation system. I have two main usecases:
To achieve these usecases, I was evaluating applying key by on input stream by product type Id in input data. This will give me data of multiple event types for that product for different marketplaces and then using a sliding event time window of lookback period say last 10mins with a sliding window of 1min and applying ProcessWindowFunction to write the correlation logic for data for last 1 mins and using other 9 mins of data for lookback and calculating price difference for interested events.
I am not fully sure if this is the best way of implementing these. Any tips/recommendations would be much appreciated!
Overall I'd say your options are:
KeyedProcessFunction
. This lower-level API offers more control and might lead to a better optimized solution. Sometimes this is also simpler, so if you find the window API is getting in your way, consider this.