[SOLVED] llegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available

I am trying to use Google PubSub. The code is relatively straight forward.

PubsubMessage message = PubsubMessage.newBuilder()
    .setData(ByteString.copyFromUtf8(messageBody))
    .putAllAttributes(messageAttributes)
    .build();
Publisher publisher = Publisher.newBuilder(TopicName.of(projectId, topicId)).build();
publisher.publish(message);

in pom.xml

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>libraries-bom</artifactId>
        <version>26.25.0</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
      ...
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-pubsub</artifactId>
    </dependency>
    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-handler</artifactId>
      <version>4.1.93.Final</version>
    </dependency>
    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-tcnative</artifactId>
      <version>2.0.61.Final</version>
    </dependency>
  </dependencies>

Please note that I only added the two netty dependencies explicitly when this failed to run on a linux server. Not shown here but I also tried adding netty-tcnative-boringssl-static dependency as well.

Running this code results in the following Exception:

Caused by: java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available

Some information about the server this is running on.

$ java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

$ nginx -v
nginx version: nginx/1.16.0

$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.5 (Maipo)
Release:    7.5
Codename:   Maipo

A lot of the commentary about how to fix this is related to the versions of netty. From: https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty

Below are known to work version combinations:
grpc-netty version  netty-handler version   netty-tcnative-boringssl-static version
1.57.x-1.58.x           4.1.93.Final            2.0.61.Final

This is the actual dependency tree (there are way more dependencies but these are the only ones that have netty explicitly mentioned:

% mvn dependency:tree

|  +- commons-beanutils:commons-beanutils:jar:1.8.0:compile
|  +- joda-time:joda-time:jar:2.8.1:compile
|  +- com.google.cloud:google-cloud-pubsub:jar:1.125.6:compile
|  |  +- io.grpc:grpc-api:jar:1.58.0:compile
|  |  +- io.grpc:grpc-stub:jar:1.58.0:runtime
|  |  +- io.grpc:grpc-protobuf:jar:1.58.0:compile
|  |  +- io.grpc:grpc-protobuf-lite:jar:1.58.0:compile
|  |  +- com.google.api:api-common:jar:2.18.0:compile
|  |  +- com.google.protobuf:protobuf-java:jar:3.24.3:compile
|  |  +- com.google.api.grpc:proto-google-common-protos:jar:2.26.0:compile
|  |  +- com.google.auth:google-auth-library-oauth2-http:jar:1.19.0:compile
|  |  +- com.google.auth:google-auth-library-credentials:jar:1.19.0:compile
|  |  +- com.google.http-client:google-http-client-gson:jar:1.43.3:compile
|  |  +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:1.107.6:compile
|  |  +- com.google.api.grpc:proto-google-iam-v1:jar:1.21.0:compile
|  |  +- com.google.api:gax:jar:2.35.0:compile
|  |  +- com.google.api:gax-grpc:jar:2.35.0:compile
|  |  +- io.grpc:grpc-alts:jar:1.58.0:compile
|  |  +- io.grpc:grpc-grpclb:jar:1.58.0:compile
|  |  +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.2:compile
|  |  +- io.grpc:grpc-auth:jar:1.58.0:compile
|  |  +- io.grpc:grpc-netty-shaded:jar:1.58.0:compile
|  |  +- io.perfmark:perfmark-api:jar:0.26.0:runtime
|  |  +- io.grpc:grpc-googleapis:jar:1.58.0:runtime
|  |  +- io.grpc:grpc-xds:jar:1.58.0:runtime
|  |  +- io.opencensus:opencensus-proto:jar:0.2.0:runtime
|  |  +- io.grpc:grpc-services:jar:1.58.0:runtime
|  |  +- com.google.re2j:re2j:jar:1.7:runtime
|  |  +- com.google.api:gax-httpjson:jar:2.35.0:compile
|  |  +- com.google.protobuf:protobuf-java-util:jar:3.24.3:compile
|  |  +- org.threeten:threetenbp:jar:1.6.8:compile
|  |  +- io.opencensus:opencensus-api:jar:0.31.1:compile
|  |  +- io.grpc:grpc-context:jar:1.58.0:compile
|  |  +- io.grpc:grpc-inprocess:jar:1.58.0:runtime
|  |  +- io.grpc:grpc-core:jar:1.58.0:compile
|  |  +- com.google.android:annotations:jar:4.1.1.4:runtime
|  |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.23:runtime
|  |  +- io.grpc:grpc-util:jar:1.58.0:runtime
|  |  +- com.google.auto.value:auto-value-annotations:jar:1.10.4:compile
|  |  +- com.google.http-client:google-http-client:jar:1.43.3:runtime
|  |  +- commons-logging:commons-logging:jar:1.2:compile
|  |  +- io.opencensus:opencensus-contrib-http-util:jar:0.31.1:runtime
|  |  \- javax.annotation:javax.annotation-api:jar:1.3.2:compile

+- io.netty:netty-handler:jar:4.1.93.Final:compile
|  +- io.netty:netty-common:jar:4.1.77.Final:compile
|  +- io.netty:netty-resolver:jar:4.1.93.Final:compile
|  +- io.netty:netty-buffer:jar:4.1.93.Final:compile
|  +- io.netty:netty-transport:jar:4.1.93.Final:compile
|  +- io.netty:netty-transport-native-unix-common:jar:4.1.93.Final:compile
|  \- io.netty:netty-codec:jar:4.1.93.Final:compile
+- io.netty:netty-tcnative:jar:2.0.61.Final:compile
|  \- io.netty:netty-tcnative-classes:jar:2.0.61.Final:compile