I'm working with latest sbt.version=1.5.7
.
My assembly.sbt
is nothing more than addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "1.1.0")
.
I have to work with a subprojects due to requirement need.
I am facing the Spark
dependencies with provided
scope similar to this post: How to work efficiently with SBT, Spark and "provided" dependencies?
As the above post said, I can manage to Compile / run
under the root project but fails when Compile / run
in the subproject.
Here's my build.sbt
detail:
val deps = Seq(
"org.apache.spark" %% "spark-sql" % "3.1.2" % "provided",
"org.apache.spark" %% "spark-core" % "3.1.2" % "provided",
"org.apache.spark" %% "spark-mllib" % "3.1.2" % "provided",
"org.apache.spark" %% "spark-avro" % "3.1.2" % "provided",
)
val analyticsFrameless =
(project in file("."))
.aggregate(sqlChoreography, impressionModelEtl)
.settings(
libraryDependencies ++= deps
)
lazy val sqlChoreography =
(project in file("sql-choreography"))
.settings(libraryDependencies ++= deps)
lazy val impressionModelEtl =
(project in file("impression-model-etl"))
// .dependsOn(analytics)
.settings(
libraryDependencies ++= deps ++ Seq(
"com.google.guava" % "guava" % "30.1.1-jre",
"io.delta" %% "delta-core" % "1.0.0",
"com.google.cloud.bigdataoss" % "gcs-connector" % "hadoop2-2.1.3"
)
)
Compile / run := Defaults
.runTask(
Compile / fullClasspath,
Compile / run / mainClass,
Compile / run / runner
)
.evaluated
impressionModelEtl / Compile / run := Defaults
.runTask(
impressionModelEtl / Compile / fullClasspath,
impressionModelEtl / Compile / run / mainClass,
impressionModelEtl / Compile / run / runner
)
.evaluated
After I execute impressionModelEtl / Compile / run
with a simple program:
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
object SparkRead {
def main(args: Array[String]): Unit = {
val spark =
SparkSession
.builder()
.master("local[*]")
.appName("SparkReadTestProvidedScope")
.getOrCreate()
spark.stop()
}
}
, it returns
[error] java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession$
[error] at SparkRead$.main(SparkRead.scala:7)
[error] at SparkRead.main(SparkRead.scala)
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[error] Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession$
[error] at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
That baffles me for days. Please help me out...Thanks so much
Finally I figured out a solution. Just separate the build.sbt
file in the parent project into its subproject.
Like in ./build.sbt
:
import Dependencies._
ThisBuild / trackInternalDependencies := TrackLevel.TrackIfMissing
ThisBuild / exportJars := true
ThisBuild / scalaVersion := "2.12.12"
ThisBuild / version := "0.0.1"
ThisBuild / Test / parallelExecution := false
ThisBuild / Test / fork := true
ThisBuild / Test / javaOptions ++= Seq(
"-Xms512M",
"-Xmx2048M",
"-XX:MaxPermSize=2048M",
"-XX:+CMSClassUnloadingEnabled"
)
val analyticsFrameless =
(project in file("."))
// .dependsOn(sqlChoreography % "compile->compile;test->test", impressionModelEtl % "compile->compile;test->test")
.settings(
libraryDependencies ++= deps
)
lazy val sqlChoreography =
(project in file("sql-choreography"))
lazy val impressionModelEtl =
(project in file("impression-model-etl"))
While in impression-model-etl
dir, create another build.sbt
file:
import Dependencies._
lazy val impressionModelEtl =
(project in file("."))
.settings(
libraryDependencies ++= deps ++ Seq(
"com.google.guava" % "guava" % "30.1.1-jre",
"io.delta" %% "delta-core" % "1.0.0",
"com.google.cloud.bigdataoss" % "gcs-connector" % "hadoop2-2.1.3"
)
// , assembly / assemblyExcludedJars := {
// val cp = (assembly / fullClasspath).value
// cp filter { _.data.getName == "org.apache.spark" }
// }
)
Compile / run := Defaults
.runTask(
Compile / fullClasspath,
Compile / run / mainClass,
Compile / run / runner
)
.evaluated
assembly / assemblyOption := (assembly / assemblyOption).value.withIncludeBin(false)
assembly / assemblyJarName := s"${name.value}_${scalaBinaryVersion.value}-${sparkVersion}_${version.value}.jar"
name := "impression"
And be sure to extract common Spark libraries to the parent project
dir, with a Dependencies.scala
file:
import sbt._
object Dependencies {
// Versions
lazy val sparkVersion = "3.1.2"
val deps = Seq(
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-mllib" % sparkVersion % "provided",
"org.apache.spark" %% "spark-avro" % sparkVersion % "provided",
...
)
}
And with all these steps done, it's normal to run Spark code locally in the subproject folder whilst setting Spark dependencies as "provided".