javascalamavenapache-spark

Spark Scala : java.lang.NoSuchFieldError: JAVA_9 at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:207)

I have a spark scala application, it uses the below versions(pasted only a section of the pom.xml) of dependencies and properties listed.

Dependencies:

  <groupId>org.apache.spark</groupId>
  <artifactId>spark-hive_2.12</artifactId>
  <version>3.3.2</version>

  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.12</artifactId>
  <version>3.3.2</version>

<dependency>
  <groupId>org.scala-lang</groupId>
  <artifactId>scala-reflect</artifactId>
  <version>2.12.10</version>
  <scope>compile</scope>
</dependency>

<dependency>
  <groupId>org.scala-lang</groupId>
  <artifactId>scala-compiler</artifactId>
  <version>2.12.10</version>
  <scope>compile</scope>
</dependency>

Properties:

<scala.major.version>2.12</scala.major.version>
<scala.test.version>3.2.18</scala.test.version>
<scala.version>2.12.18</scala.version>

<maven.assembly.plugin.verion>3.3.0</maven.assembly.plugin.verion>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.deploy.plugin.version>2.8.2</maven.deploy.plugin.version>
<maven.properties.plugin.verion>1.0.0</maven.properties.plugin.verion>
<maven.release.plugin.version>3.0.0</maven.release.plugin.version>
<maven.scala.plugin.version>4.5.6</maven.scala.plugin.version>
<maven.shade.plugin.version>3.5.2</maven.shade.plugin.version>
<maven.site.plugin.version>3.12.1</maven.site.plugin.version>
<maven.source.plugin.version>3.2.1</maven.source.plugin.version>

My local setup has java=1.8.0_202 and mvn=3.9.6

I'm able to successfully compile the src/main and src/test --> "mvn test-compile".

When I run "mvn install", the unit tests start failing with the following error.

*** RUN ABORTED ***
An exception or error caused a run to abort: JAVA_9 
  java.lang.NoSuchFieldError: JAVA_9
  at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:207)
  at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
  at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:114)
  at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:353)
  at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:290)
  at org.apache.spark.SparkEnv$.create(SparkEnv.scala:339)
  at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
  at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)
  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2714)

Kindly help me fix this. Also, I don't understand why does the control go into the "if" block of "org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:207)" as the application is only compiled with Java 8.

Solution

  • The issue is not due to the code going into the if block. It is because it cannot evaluate the if statement.

    NoSuchFieldError is most of the time caused by having mixed different versions of the same library.

    In this case, it's related to org.apache.commons.lang3.JavaVersion used at runtime not having the JAVA_9 attribute. Likely because you're pulling a too old version of Apache Commons Lang 3 library whereas Spark expects a more recent one.

    You can run mvn dependency:tree to check for inconsistencies in the versions pulled directly or transitively.