When stopping the CosmosDB ChangeFeedProcessor, does the StopAsync method waits for any ongoing processing of changes (in a ChangeFeedHandler) to complete before returning ? In other words, when StopAsync completes, are we sure that no ChangeFeedHandler is running or scheduled to run ?
Also, does calling StopAsync set the ChangeFeedHandler CancellationToken if processing is ongoing ?
And finally, in the ChangeFeedHandler, what is the best way to handle the CancellationToken to make sure that any processed documents are not processed again while making sure that the remaining documents will be resent for processing when the CangeFeedProcessor is restarted ?
My goal is to make sure that if the server is gracefully stopped, then later restarted, no changes will be lost and no changes we be processed twice.
NOTE: The StopAsync documentation does not say anything...
StopAsync signals the Load Balancer (grabbing new leases) and Controller that they should stop and waits until the inflight processing completes. It also signals the Renewer and Processor for each lease. This is all done through CancellationToken signalling (which is the same CancellationToken your ChangeFeedHandler is receiving).
This however might not mean that a current batch of changes completed processing, it can still fail due to, for example, an exception in the Handler code.
References:
no changes we be processed twice
This is impossible. Change Feed Processor has an "at least once" guarantee, not a "only once guarantee".
So even if this scenario (stopping) would make sure to finish all inflight processing, there would still be cases where there can be double deliveries or double processing of the same event.