ALPN negotiation error from gRPC Server on java gRPC client using envoy proxy in the middle

I have a java (java-11) gRPC client streamer trying to talk with a golang gRPC server, with tls and envoy proxy in the middle.

And I'm getting

UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible protocol

error from the ServerResponseObserver.

Full stacktrace:

12/05/22 18:29:29 (798) grpc-default-executor-1 SEVERE *** ERROR *** server respond with onError: 
io.grpc.StatusRuntimeException: UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible protocol
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
    at io.grpc.Status.asRuntimeException(Status.java:535)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:468)
    at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:432)
    at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:465)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)

The calls do not reach the gRPC server and seem like they won't pass the envoy proxy.
envoy trace logs show a few ssl errors such as "ssl error occurred while read: WANT_READ"

configuration of the java gRPC client streamer:

    ChannelCredentials credentials;
    credentials = TlsChannelCredentials.newBuilder()
            .trustManager(new File("<path-to-file>/cacert.pem"))
            .build();
    originChannel = Grpc.newChannelBuilderForAddress(host, port, credentials).build();
    ClientInterceptor interceptor = new HeaderClientInterceptor();
    Channel channel = ClientInterceptors.intercept(originChannel, interceptor);

    stub = LogStreamerGrpc.newStub(channel);

Solution

Apparently, when using a proxy one need to ensure the proxy is configured correctly with h2 enabled for ALPN

from envoy documantation alpn_protocols:

alpn_protocols:
(repeated string) Supplies the list of ALPN protocols that the listener should expose. In practice this is likely to be set to one of two values (see the codec_type parameter in the HTTP connection manager for more information):

“h2,http/1.1” If the listener is going to support both HTTP/2 and HTTP/1.1.

“http/1.1” If the listener is only going to support HTTP/1.1.

There is no default for this parameter. If empty, Envoy will not expose ALPN.

For me, supporting h2 was enough.
Here is the configuration I added to envoy.yaml running on go under in order to make it work:

transport_socket:
  common_tls_context:
    alpn_protocols: h2