I am working with an Apache Flink application that reads data streams from Kafka. The application processes the streams, creating objects from the data and calculating the processing time by marking the start time at object creation and the end time after performing a series of transformations including keyBy and some calculations.
In most cases, the processing times recorded are between 0-1 ms. However, I've noticed that some results show an unusually high processing time of over 10 ms. This behavior seems to be intermittent and varies between runs, despite the following consistent setup:
What could be causing these intermittent spikes in processing time? How can I diagnose the underlying issue? Or is this fluctuation normal?
That sort of variation in latency isn't surprising.
First thing that comes to mind is garbage collection. If you haven't already done so, it might help to use the latest Flink release with Java 17 or 21, with ZGC.
There are many other configuration details that could play a factor too. Without knowing a lot more about your application and environment it's hard to know where to start.