.netazureazure-web-app-servicecontainersfreeze

.Net API Azure App Service hangs on startup when under load

I am experiencing an issue with my .NET API hosted in Azure App Service. The application starts without any issues under normal load, but when the load increases, the application hangs during startup. Running on a single instance of P2v3.

Our startup time takes between 5-10 seconds, we are experiencing roughly 100req/s when it fails to startup.

Once it's stuck, the application isn't able to recover on it's own however looking at the logs, the application isn't receiving any requests. The requests are all timing out.

Looking in logs I can see some common logs that seem to occurring:

Failed to forward request to http://xxx.xxx.xxx.3:8080. Encountered a System.Threading.Tasks.TaskCanceledException exception after 300002.419ms with message: The request was cancelled due to the configured HttpClient.Timeout of 300 seconds elapsing.. Check application logs to verify the application is properly handling HTTP traffic.

However I'm not seeing the requests in any other logs, even though application is now listening on... and we are seeing a mixture of 503's, 504's, 500's.

You can see here for example there was a 15 minute gap where nothing was happening except for those timeouts and the logs are just empty.

Empty Logs

We've been recovering by re-deploying so far which deploys into a cold slot and once it's warmed up swaps it over. enter image description here

I have been able to replicate it locally, however after bringing startup down from ~3 minutes, to 10 seconds locally it seems to handle everything okay now.

UPDATE: The solution ended up being to run a minimum of 3 app service instances, even bringing the startup-up down to ~10 seconds didn't work in a high-load environment. Since going to 3 instances, we haven't seen this issue come up again.

Solution

  • In order to improve the performance of your Web app and improve its startup, Refer the steps below:-

    Visit Diagnose and solve problems > Risk alerts > Availability > View more details

    If your Web App is running on 2 and less instance around 50% downtime is expected as the instances will be upgraded. Consider upgrading your web app to 3 instances with auto scale or by manually scaling your web app. With 3 instances 1 instance will be in different upgrade domain thus it will improve your performance and avoid downtime.

    enter image description here

    Scale out your Web app, choose any one option out of the three.

    Reference- How to enable automatic scaling - Azure App Service | Microsoft Learn

    Always ready instances will help you avoid cold starts, As the Web app instances will be warmed up and ready to serve requests whenever required.

    enter image description here

    Also, In Diagnose and solve problems > Availability and Performance > Web App Slow and High CPU Analysis to get insight on your Web app performance.

    enter image description here

    enter image description here

    Also, Load your web app in the browser and check the Log stream, Log stream will give all the Start up logs in detail:-

    enter image description here