azure-functionsazure-durable-functions

Azure (Durable) Functions control concurrency multiple instances fan-out


I'm running an .NET 7 (isolated) function app on an App Service Plan with 10 instances (scale-out). There is a durable orchestrator function performing a fan-out of about 100 function invocations (simultaneously). Due to the cpu/memory requirements of running these functions, I can only run 2 invocations on the same instance. So a total of 20 invocations concurrently over these 10 instances.

The fan-out starts with using all instances, but after a while I only see 1 or 2 instances picking up new invocations, while other instances are idling.

So what I think happens is that the 100 invocations are divided amongst the 10 instances in advance, and some instances get (by random chance) 10 short invocations, where others might get 10 long invocations.

I tried the following (simplified) configuration to control this behaviour, but now I'm only seeing 2 instances (in total) doing the work, and the other 8 idling without having picked up any work.

// host.json
{
    "version": "2.0",
    "extensions": {
        "durableTask": {
            "storageProvider": {
                "controlQueueBatchSize": 1,
                "controlQueueBufferThreshold": 1
            },
            "maxConcurrentActivityFunctions": 2
        }
    },
    "functionTimeout": "-1"
}

How can I achieve the following behaviour: process all 100 invocations as quickly as possible by using all 10 instances?


Solution

  • I found that the scale-out was only considering booted instances. As one instance was handling the inbound triggering HTTP request, there would only be a single instance. One way to get the other instances to be considered is by turning on 'Always On'. I deliberately chose not to use 'Always On' as for this use-case I don't want the instances running all the time. I now have them running all the time, while they are only utilised for a fraction of the day.