service-fabric-statefulazure-service-fabric

Force return from RunAsync() via decision made in web API Controller in Service Fabric

This thread is highly relevant to the scenario when you see exception thrown such as FabricObjectClosedException.

The question is for specific scenario when you see the above exception while en-queue using StateManager commit in web api controller upon http-request.

To give more background story, FabricObjectCloseException is thrown when the Service Fabric object is currently in a closed state due to one of reasons (here):

  1. The Service Fabric object is being deleted.
  2. The Service Fabric object is not reachable due to a failover.

If the above exception were thrown within the RunAsync() while loop, the recommended practice could be return out of the RunAsync() method, which invokes restart of the service based on the link.

What if the exception was thrown while trying to use reliable state manager in web API controller? How can the RunAsync() of the stateful service get out of the while loop in this case?

Solution

  • I am not 100% sure that I understand the full picture but I can think of the situations below:

    You have:

    1. Web API controller that is exposed as ServiceReplicaListener to serve incoming requests
    2. Some background / continues work done in the RunAsync method

    The first thing here is to check whether CancellationToken passed to RunAsync is set when FabricObjectCloseException is thrown.

    As far as I understand FabricObjectCloseException is thrown when replica is being shutdown (here I once had described the steps Service Fabric performs when service is shutdown) by Cluster Manager or by some other reasons. If this assumption is true then CancellationToken should be set.

    If passed CancellationToken isn't set - then I could advice you to:

    1. Create a simple service that wraps CancellationTokenSource
    2. Create instance of CancellationTokenSource inside the constructor of StatefulService class and initialize a service from #1.
    3. Using dependency injection, inject this service in Web API controller and signal cancellation when FabricObjectCloseException occurs.
    4. In RunAsync, check the cancellation status of both: passed CancellationToken and custom service.

    P.S. I am not sure about what kind of work you do in the RunAsync method but I just want to mention that it is OK to return from RunAsync when the background part of the work is done - or to even not override it when there is no background work.