I'm using Python to access Databricks through databricks-connect. Behind the wall, this uses spark which is indeed java based so in order to use this, I need java. The JDK has been downloaded (version 14), set as JAVA_HOME env but when I run the code, I get the following error:
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @bccb269
This is the code where it crashes
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
I googled but couldn't find this error, I don't think it's the first time occurring to me. Any ideas what this error means and how can I fix it?
Databricks Runtimes and databricks-connect won't work with Java 14. Only DBR 10.x have an experimental support for Java 11, but I doubt that it's supported for databricks-connect.
You need to install Java 8 to use databricks-connect.