javakubernetesapache-flinkk3sk3d

JcaPEMKeyConverter is provided by BouncyCastle, an optional dependency. To use support for EC Keys you must explicitly add dependency to classpath

I have a simple Flink streaming app. It runs well in a cluster created by start-cluster.sh command.

Now based on the Flink tutorial, I hope to deploy it in application mode natively in a Kubernetes cluster created by k3d on macOS.

First, I created a cluster by k3d cluster create dev.

Here is my Dockerfile:

FROM flink
RUN mkdir -p $FLINK_HOME/usrlib
COPY target/streaming-0.1.jar $FLINK_HOME/usrlib/streaming-0.1.jar

I built and pushed it to Docker Hub.

My cluster name is k3d-dev, so I ran

flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=k3d-dev \
    -Dkubernetes.container.image=hongbomiao/my-flink-xxx:latest \
    local:///opt/flink/usrlib/streaming-0.1.jar

However, I got error:

 The program finished with the following exception:

io.fabric8.kubernetes.client.KubernetesClientException: JcaPEMKeyConverter is provided by BouncyCastle, an optional dependency. To use support for EC Keys you must explicitly add this dependency to classpath.
    at io.fabric8.kubernetes.client.internal.CertUtils.handleECKey(CertUtils.java:161)
    at io.fabric8.kubernetes.client.internal.CertUtils.loadKey(CertUtils.java:131)
    at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:111)
    at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:243)
    at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
    at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:82)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:62)
    at io.fabric8.kubernetes.client.BaseClient.<init>(BaseClient.java:51)
    at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:105)
    at org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:102)
    at org.apache.flink.kubernetes.KubernetesClusterClientFactory.createClusterDescriptor(KubernetesClusterClientFactory.java:61)
    at org.apache.flink.kubernetes.KubernetesClusterClientFactory.createClusterDescriptor(KubernetesClusterClientFactory.java:39)
    at org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:63)
    at org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:213)
    at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1057)
    at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
    at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)

After reading

I added

<dependency>
    <groupId>org.bouncycastle</groupId>
    <artifactId>bcpkix-jdk15on</artifactId>
    <version>1.69</version>
</dependency>
<dependency>
    <groupId>org.bouncycastle</groupId>
    <artifactId>bcprov-jdk15on</artifactId>
    <version>1.69</version>
</dependency>
<dependency>
    <groupId>org.bouncycastle</groupId>
    <artifactId>bcprov-ext-jdk15on</artifactId>
    <version>1.69</version>
</dependency>

to my pom.xml file. I built and pushed to Docker Hub again.

When I ran the Flink command above, I still got same error. Any idea? Thanks!

UPDATE 1:

Besides above pom.xml change, I manually downloaded those 3 jars and changed my Dockerfile to

FROM flink
COPY lib/* $FLINK_HOME/lib
RUN mkdir -p $FLINK_HOME/usrlib
COPY target/streaming-0.1.jar $FLINK_HOME/usrlib/streaming-0.1.jar

and tried again, but still same error.

I can confirm the 3 jar files bcpkix-jdk15on-1.69.jar, bcprov-ext-jdk15on-1.69.jar, bcprov-jdk15on-1.69.jar are in the docker image:

āžœ docker run -it 6c48af48db55c334003a307d1ef7a5fc5181f389613284b66b5cb97588b9708d sh

$ cd lib && ls
bcpkix-jdk15on-1.69.jar      flink-dist_2.12-1.13.2.jar     flink-table_2.12-1.13.2.jar  log4j-slf4j-impl-2.12.1.jar
bcprov-ext-jdk15on-1.69.jar  flink-json-1.13.2.jar      log4j-1.2-api-2.12.1.jar
bcprov-jdk15on-1.69.jar      flink-shaded-zookeeper-3.4.14.jar  log4j-api-2.12.1.jar
flink-csv-1.13.2.jar         flink-table-blink_2.12-1.13.2.jar  log4j-core-2.12.1.jar
$ cd ../usrlib && ls
streaming-0.1.jar

UPDATE 2:

I tried to start the session mode by

/usr/local/Cellar/apache-flink/1.13.1/libexec/bin/kubernetes-session.sh

but still got same error. So now I can confirm when I was using application mode before, the issue is not related my Docker image.

I have those jars are located at ~/.m2 on my machine:

enter image description here

Did I miss any other jars?

Also, I found the error only happens to the cluster created by k3d/k3s, but not minikube.

Solution

  • TL;DR

    Download latest version of bcprov-jdk15on and bcpkix-jdk15on jar files from

    Then move to your apache-flink folder. In my case, it is

    /usr/local/Cellar/apache-flink/{version}/libexec/lib

    Then you are good to go!

    How I Found Out

    After checking the code of

    The first script is pointing to the second script, and the second script has

    # ...
CC_CLASSPATH=`manglePathList $(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPATHS`

# ...
"$JAVA_RUN" $JVM_ARGS -classpath "$CC_CLASSPATH" $log_setting org.apache.flink.kubernetes.cli.KubernetesSessionCli "$@"

    I added echo $CC_CLASSPATH, and printed out the classpath.

    In my case, it is at /usr/local/Cellar/apache-flink/1.13.1/libexec/lib.

    After I put bcprov-jdk15on-1.69.jar and bcpkix-jdk15on-1.69.jar in the folder above, Flink can be deployed to k3s (k3d) now in both session and application modes.