How to change Provided dependencies when using Intellij with SBT?
My build.sbt looks like this:
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-aws" % sparkVersion % Provided,
"org.apache.spark" %% "spark-core" % sparkVersion % Provided,
"org.apache.spark" %% "spark-sql" % sparkVersion % Provided,
"org.scala-lang" % "scala-library" % scalaVersion.value % Provided
)
While running my application from Intellij, I get NoClassDefFoundError exceptions because it cannot find spark libraries. So when using Intellij, I need:
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-aws" % sparkVersion,
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.scala-lang" % "scala-library" % scalaVersion.value % Provided
)
But that causes the final flat jar to be very big.
How to have different list of Provided dependencies depending whether using Intellij or not?
Solution
The solution for my use case was to check IntelliJ Add dependencies with "provided" scope to classpath option. Found it here:
How to work efficiently with SBT, Spark and "provided" dependencies?
Since UI has changed, here are the screenshots for Intellij 2021.2.3:
After selecting it, you should see:
- How to override setup and cleanup methods in spark map function
- Scala: How to interpret foldLeft
- How do I convert a Map[Int, Any] to a SortedMap in Scala? Or a TreeMap?
- Take default value if not present
- How do I archive multiple files into a .zip file using scala?
- Flatten nested json in Scala Spark Dataframe
- Are there any methods included in Scala to convert tuples to lists?
- Retrieving key(s) from a Scala map using a value
- private[this] vs private
- zio-http and Tapir: provide layers globally
- Read pdf file in apache spark dataframes
- How to profile methods in Scala?
- Typesafe config: get as map
- How to fork the jvm for each test in sbt
- Can a scala3 macro introduce an implicit value around an existing block?
- Scala: java.lang.NoClassDefFoundError: Could not initialize class sbt.internal.parser.SbtParser
- Writing to a delta table spark 3.5.3 delta lake 3.2.0
- Clarifying cross version behavior in Scala
- How to read PDF files and xml files in Apache Spark scala?
- Spark UDF Null handling
- Scala Akka HTTP problem adding multiple segment variables to API URL Routes
- How to convert map to dataframe?
- Is there a way to create a RSocket "forRequestStream" and return it as a Pekko/Akka Sink without using the deprecated FluxProcessor?
- Spark Scala : java.lang.NoSuchFieldError: JAVA_9 at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:207)
- creating parallel processing of parenthesis balancer
- Why "avoid method overloading"?
- Where is the documentation for scala.language.postfixOps?
- In Scala 3, how to write a term of which type signature is a match type?
- Chronomorphisms in Scala
- Remove all records which are duplicate in spark dataframe