Let's suppose I have a project 'internal-library' with this build.sbt:
organization := "com.foo"
name := "internal-library"
version := "1.0.0"
scalaVersion := "2.13.13"
...other stuff...
libraryDependencies += "com.mysql" % "mysql-connector-j" % "8.0.32"
libraryDependencies += "com.clickhouse" % "clickhouse-jdbc" % "0.4.6"
...other dependencies...
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "3.5.1" % "provided",
"org.apache.spark" %% "spark-sql" % "3.5.1" % "provided"
)
The project 'internal-library' contains code and dependencies that are useful for many other projects and it's published in local repository with command sbt clean publishLocal.
Now, let's suppose I have a project 'abc' with this build.sbt:
organization := "com.foo"
name := "abc"
version := "1.0.0"
scalaVersion := "2.13.13"
...other stuff...
libraryDependencies += "com.foo" %% "internal-project" % "1.0.0"
All the dependencies declared in build.sbt of 'internal-project' are also visible and usable for compilation to project 'abc' except the ones declared as provided.
At this point I'm forced to add also in the sub project the spark library dependencies, but I would like to have them declared in only one place and they should be declared as provided because spark cluster already owns spark's jars and it doesn't need them in the fat jar of project 'abc'.
I've found the solution. The build.sbt of the child ('abc' project) should be:
organization := "com.foo"
name := "abc"
version := "1.0.0"
scalaVersion := "2.13.13"
...other stuff...
libraryDependencies += "com.foo" %% "internal-project" % "1.0.0" % "compile->compile;provided->provided"
In this way you can avoid to include provided dependencies already declared in parent project ('internal-library').
I've got inspiration for this solution by this answer on stack overflow and by sbt documentation.