"Http2ConnectionException: HTTP/2 error code 'PROTOCOL_ERROR' (0x1)" error when enabling VNet integration for Azure Function App

When enabling VNet integration on an Azure Function App to communicate with outbound internal services over private endpoints, an HTTP 502 Bad Gateway or HTTP 504 Gateway Timeout error occurs. If we disable VNet integration, the function app starts working normally.

Here is a diagram of our configuration:

In the logs, we see the following:

2022-10-14T15:04:59.443 [Information] Host initialized (1052ms)
2022-10-14T15:04:59.458 [Information] Host started (1107ms)
2022-10-14T15:04:59.459 [Information] Job host started
2022-10-14T15:04:59.500 [Debug] Debug file watch initialized.
2022-10-14T15:04:59.511 [Debug] Diagnostic file watch initialized.
2022-10-14T15:04:59.512 [Debug] Hosting started
2022-10-14T15:05:02.400 [Error] Unhandled exception. System.IO.IOException: The request was aborted.
2022-10-14T15:05:02.413 [Error] ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).
2022-10-14T15:05:02.418 [Error] at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode)
2022-10-14T15:05:02.425 [Information] at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
2022-10-14T15:05:02.425 [Information] at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
2022-10-14T15:05:02.425 [Information] --- End of inner exception stack trace ---
2022-10-14T15:05:02.425 [Information] at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
2022-10-14T15:05:02.425 [Information] at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
2022-10-14T15:05:02.425 [Information] at Grpc.Net.Client.StreamExtensions.WriteMessageAsync[TMessage](Stream stream, GrpcCall call, TMessage message, Action`2 serializer, CallOptions callOptions)
2022-10-14T15:05:02.425 [Information] at Grpc.Net.Client.Internal.HttpContentClientStreamWriter`2.WriteAsyncCore[TState](Func`5 writeFunc, TState state)
2022-10-14T15:05:02.425 [Information] at Microsoft.Azure.Functions.Worker.GrpcWorker.SendStartStreamMessageAsync(IClientStreamWriter`1 requestStream) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 84
2022-10-14T15:05:02.426 [Information] at Microsoft.Azure.Functions.Worker.GrpcWorker.StartAsync(CancellationToken token) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 61
2022-10-14T15:05:02.426 [Information] at Microsoft.Azure.Functions.Worker.WorkerHostedService.StartAsync(CancellationToken cancellationToken) in D:\a\1\s\src\DotNetWorker.Core\WorkerHostedService.cs:line 25
2022-10-14T15:05:02.426 [Information] at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
2022-10-14T15:05:02.426 [Information] at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2022-10-14T15:05:02.426 [Information] at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2022-10-14T15:05:02.426 [Information] at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
2022-10-14T15:05:02.426 [Information] at MyCompany.MyApp.Api.Program.Main() in /home/vsts/work/1/s/src/Api/Program.cs:line 14
2022-10-14T15:05:02.681 [Debug] Handling WorkerErrorEvent for runtime:dotnet-isolated, workerId:dotnet-isolated. Failed with: Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException: dotnet exited with code 134 (0x86)
---> System.Exception: Unhandled exception. System.IO.IOException: The request was aborted., ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).,   at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode)
   --- End of inner exception stack trace ---
2022-10-14T15:05:02.681 [Debug] Attempting to dispose webhost or jobhost channel for workerId: '6778e415-98a9-4167-8bf6-582416320e64', runtime: 'dotnet-isolated'
2022-10-14T15:05:02.681 [Debug] Disposing language worker channel with id:6778e415-98a9-4167-8bf6-582416320e64

What can cause this behavior? We have tried using the latest .NET Framework. There are also custom DNS servers and route tables configured in this environment, but we leverage the same infrastructure we use for all other private endpoint configurations, which work without issues.

The cause appears to be an unidentified and unresolved defect from the Azure App Service side. There is no known official solution. However, we were able to work around the solution by re-engineering the application to switch the .NET worker process from 'isolated' to 'in-process.' Here is a longer description from our development team:

Earlier, the dotnet core application (.Net 6) was running with runtime set to 'isolated', i.e., it was using Microsoft.Azure.Functions.Worker.* dependencies in the code. This was throwing runtime exception at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel whenever we hosted the function app to Azure.

We switched the runtime to 'in-proc', by removing all Microsoft.Azure.Functions.Worker.* dependencies - This involved some changes to the application code because things like middlewares were not automatically supported now. But removing those dependencies and changing runtime did the trick. We did not see any runtime errors in Azure.

Everything else like storage-account, platform (Linux) was kept same. Only change was in the application runtime and the Microsoft worker dependency.

Thanks.