I have an input stream, read from a database, with measurements from different devices. The events recorded in the database are not in chronological order but usually arrive in a 2 minute window. however some devices can send data with a timestamp several days in the past.
How can I process data for a device that is days behind the rest of the data when I've inserted CTIs in the input stream to deal with the "normal" data that's just a few minutes old at most?
Is it possible to split the input stream into 1 stream per device before I insert CTIs so the "older" stream will have CTIs that are independent of the other input streams?
Thanks in advance.
Good questions.
StreamInsight can handle late arriving events, you just need to understand that CTI events advance application time. Meaning that if a point event arrives with a start time earlier than the last CTI event, it will be dropped. You'll need to configure your advance time settings to delay the CTI event to allow for late arriving events. More on that can be found here: Advancing Application Time
As far as the best way to process data for a device that is days behind, you would probably be better off just recording the events as they come in and then re-playing the events after all the data has arrived. Depending on your needs, the start times of the events could always be the timestamp of when they were received and you could have a property on your events for the original timestamp.
You can have multiple streams that perform the same query logic at different application times, but they would have to have separate CTI events.