gRPC UnsupportedAddressTypeException, but only when packaged with shadowJar

I have a simple gRPC client packaged in a jar by Gradle with shadowJar. When I run the main() with IntelliJ, the RPC is sent successfully. When I run it with java -jar, I get an exception:

Update: I think I've determined that the same set of classes is provided by both IntelliJ and ./gradlew shadowJar, but the order is different. I suspect this causes the divergent behavior but I don't really understand why, or how to control the order of the classpath in either case. It would be a big help if I could figure out which classes are actually relevant here. The full expansion include 19k class files.

Exception in thread "main" io.grpc.StatusException: UNKNOWN
        at io.grpc.Status.asException(Status.java:550)
        at io.grpc.kotlin.ClientCalls$rpcImpl$1$1$1.onClose(ClientCalls.kt:296)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.nio.channels.UnsupportedAddressTypeException
        at java.base/sun.nio.ch.Net.checkAddress(Net.java:146)
        at java.base/sun.nio.ch.Net.checkAddress(Net.java:157)
        at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816)
        at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839)
        at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:91)
        at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:88)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:88)
        at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:322)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:248)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
        at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:54)
        at io.grpc.netty.WriteBufferingAndExceptionHandler.connect(WriteBufferingAndExceptionHandler.java:157)
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at io.netty.channel.AbstractChannelHandlerContext.access$1000(AbstractChannelHandlerContext.java:61)
        at io.netty.channel.AbstractChannelHandlerContext$9.run(AbstractChannelHandlerContext.java:538)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        ... 1 more

my client (which does not appear to be the issue):

fun main() {
    runBlocking {
        SampleClient().hello("hello")
    }
}
class SampleClient() : Closeable {
    private val channel = ManagedChannelBuilder.forAddress("localhost", StationController.PORT).usePlaintext().build()
    private val stub = StationServiceGrpcKt.StationServiceCoroutineStub(channel)
    suspend fun hello(msg: String) {
        val request = helloRequest { message = msg }
        stub.hello(request)
    }
    override fun close() {
        channel.shutdown().awaitTermination(5, TimeUnit.SECONDS)
    }
}

Running with java -jar build/libs/test-0.1-all.jar fails with the above exception. Running with java -classpath <very long classpath> com.test.MainKt succeeds.

I tried to minimize my gradle.build.kts that still shows this behavior:

import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
import com.google.protobuf.gradle.*

plugins {
    kotlin("jvm") version "1.7.10"
    id("com.github.johnrengelman.shadow") version "6.0.0"
    id("com.google.protobuf") version "0.8.19"
}

// These address this error:
// 'compileJava' task (current target is 17) and 'compileKotlin' task (current target is 1.8) jvm target compatibility
// should be set to the same Java version.
java { sourceCompatibility = JavaVersion.VERSION_17; targetCompatibility = JavaVersion.VERSION_17 }
tasks.withType<KotlinCompile> { kotlinOptions { jvmTarget = "17" } }

group = "com.test"
version = "0.1"

repositories { mavenCentral() }

dependencies {
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.3")
    implementation("io.grpc:grpc-protobuf:1.48.1")
    implementation("io.grpc:grpc-kotlin-stub:1.3.0")

    // The rpc fails only when I run it from the fat jar. running from intellij works. the stacktrace is all in netty,
    // but when I compile without netty there is no netty in the jar, making me think there is no transitive dependency
    // on netty. it fails with both netty and netty-shaded, but with different stack traces.
    // with netty-shaded, it says Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: connect(..) failed: Address family not supported by protocol: /localhost:8980
    // with netty, it just says Caused by: java.nio.channels.UnsupportedAddressTypeException
    implementation("io.grpc:grpc-netty:1.48.1")

    implementation("com.google.protobuf:protobuf-kotlin:3.21.4")
}

// for gRPC
protobuf {
    protoc { artifact = "com.google.protobuf:protoc:3.21.4" }
    plugins {
        // We only need this for Kotlin, but it doesn't fully link without the java sources.
        id("grpc") { artifact = "io.grpc:protoc-gen-grpc-java:1.48.1" }
        id("grpckt") { artifact = "io.grpc:protoc-gen-grpc-kotlin:1.3.0:jdk8@jar" }
    }
    generateProtoTasks {
        all().forEach {
            it.plugins { id("grpc"); id("grpckt") }
            // The plugins above (I think) generate the Java proto code, but not the extra Kotlin stuff like DSLs.
            // Including this adds a /kotlin folder to the generated code which contains all the extra stuff.
            it.builtins { id("kotlin") }
        }
    }
}


sourceSets {
    main {
        java {
            // These are the default generated source dirs, but they have to be added manually currently:
            // https://github.com/google/protobuf-gradle-plugin/issues/109
            // The plugin generates both a grpc dir with Java files, and grpckt with Kotlin files. I believe the grpckt
            // contents includes all the functionality of the grpc (java) contents, but I'm not positive. The kotlin
            // files might just be stubs for the java version? so maybe this compiles but doesn't run?
            srcDirs("build/generated/source/proto/main/grpc")
            srcDirs("build/generated/source/proto/main/grpckt")
            // This contains the Java classes like the request and response messages.
            srcDirs("build/generated/source/proto/main/java")
            // This contains the Kotlin extensions to those java classes, like DSLs.
            srcDirs("build/generated/source/proto/main/kotlin")
        }
    }
}

tasks.jar { manifest.attributes["Main-Class"] = "com.test.MainKt" }

I'm at a loss for how to proceed with debugging this.

A friend found the answer. This addition to build.gradle.kts ended up being all it took to make it work:

tasks.named("shadowJar", ShadowJar::class) {
    mergeServiceFiles()
}

See shadowJar's documentation about what's happening here. Summary: "Java libraries often contain service descriptor files in the META-INF/services directory of the JAR. A service descriptor typically contains a line delimited list of classes that are supported for a particular service. At runtime, this file is read and used to configure library or application behavior.

Multiple dependencies may use the same service descriptor file name. In this case, it is generally desired to merge the content of each instance of the file into a single output file." mergeServiceFiles() does this.