We have a dotnet api running on an Azure App Service that is routed through an Application Gateway. For one particular api call, we are seeing strange behavior occurring.
When the api is used as intended, with a frontend ui, the response the frontend receives is a 504 error after 4 minutes (which I believe is the time limit set in Azure for App Services). This also occurs if you call the api directly using something like Postman. Application Insights however, is reporting that the return status code for this particular operation is actually a 499 error code.
However, if you bypass the custom domain set for this api and call the App Service uri directly, this very same response with the same data returns after a little over 2-3 minutes (well below the app service max) successfully as we want without either a 499 or 504 error status code.
This has lead me to believe that the issue must be coming from the Application Gateway itself. However, in the backend settings for this particular application, the timeout is set to 600 seconds, so I am unsure where else this sort of limit is taking place. Is there another setting in the App Gateway I should be looking at that would cause this problem?
The issue is caused by a timeout mismatch between the client, Application Gateway, and the App Service,
Azure Application Gateway has a default timeout of 230 seconds, if your backend takes longer than this to respond, the gateway closes the connection.
This causes a 499 error in Application Insights (because the gateway canceled the request) and a 504 error on the frontend (since the client didn’t get a response in time).
To resolve the issue,
Increase the timeout value in the Azure Application Gateway’s HTTP settings,
Go to Azure portal - > Application Gateway - > HTTP settings - > Your setting -> Request Timeout.
Increase it to a value higher than your API’s max expected execution time, e.g., set to 600 seconds.
Test with Postman or curl using extended timeouts,
Refer this doc to know more about curl timeout options.
curl -m 600 https://your-api-endpoint
If possible, use Azure Front Door Instead of App Gateway, as it handles long-running requests better in some cases.