I'm facing some deduplicate errors like below, which indicate that some of my dependencies import other dependencies which contains files with the same path name. Since they have the same path, they cannot be included together in the jar file that I'm trying to create with sbt-assembly.
I understand that the clean way to fix it is by fixing my dependencies. The conflicting dependencies from the example below seem to be reactive-streams and reactive-streams-flow-adapters, but I'm not sure what they are and where they come from. How can I find which of my dependencies are importing them?
If I can figure that out, how can I fix it? Is there a way other than just removing one of them?
An example of deduplicate errors:
[error] (assembly) deduplicate: different file contents found in the following:
[error] /home/guillaume/.cache/coursier/v1/https/repo1.maven.org/maven2/org/reactivestreams/reactive-streams-flow-adapters/1.0.2/reactive-streams-flow-adapters-1.0.2.jar:org/reactivestreams/FlowAdapters$FlowPublisherFromReactive.class
[error] /home/guillaume/.cache/coursier/v1/https/repo1.maven.org/maven2/org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.jar:org/reactivestreams/FlowAdapters$FlowPublisherFromReactive.class
Here is a list of my dependencies, if it helps:
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.12.2",
libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test",
libraryDependencies += "com.softwaremill.sttp.client3" %% "core" % "3.3.6",
libraryDependencies += "com.softwaremill.sttp.client3" %% "httpclient-backend-zio" % "3.3.6",
libraryDependencies += "dev.zio" %% "zio" % "1.0.9",
// libraryDependencies += "dev.zio" %% "zio-streams" % "1.0.9",
libraryDependencies += "org.apache.commons" % "commons-compress" % "1.20",
libraryDependencies += "org.scalactic" %% "scalactic" % "3.2.9",
libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.9" % "test",
libraryDependencies += ("tech.sparse" %% "toml-scala" % "0.2.2").cross(CrossVersion.for3Use2_13),
I'm not familiar with Reactive Streams, but it looks like they made some changes during the patch version and moved classes around in 1.0.3. See Reactive Streams 1.0.3 is here!
When we released 1.0.2, we shipped a compatibility/conversion library to seamlessly convert between the
java.util.concurrent.Flowand theorg.reactivestreamsnamespaces—in 1.0.3 these adapters are instead included in the main 1.0.3 jar.
How can I find which of my dependencies are importing them?
If you're using the latest sbt 1.5.4, it has Johannes's dependency graph plugin built-in by default, so you can run dependencyTree from sbt shell:
sbt:root> dependencyTree
[info] root:root_2.13:0.1.0-SNAPSHOT [S]
....
[info] +-com.softwaremill.sttp.client3:httpclient-backend-zio_2.13:3.3.6 [S]
[info] | +-com.softwaremill.sttp.client3:httpclient-backend_2.13:3.3.6 [S]
[info] | | +-com.softwaremill.sttp.client3:core_2.13:3.3.6 [S]
[info] | | | +-com.softwaremill.sttp.model:core_2.13:1.4.7 [S]
[info] | | | +-com.softwaremill.sttp.shared:core_2.13:1.2.5 [S]
[info] | | | +-com.softwaremill.sttp.shared:ws_2.13:1.2.5 [S]
[info] | | | +-com.softwaremill.sttp.model:core_2.13:1.4.7 [S]
[info] | | | +-com.softwaremill.sttp.shared:core_2.13:1.2.5 [S]
[info] | | |
[info] | | +-org.reactivestreams:reactive-streams-flow-adapters:1.0.2
[info] | | +-org.reactivestreams:reactive-streams:1.0.2 (evicted by: 1.0.3)
[info] | | +-org.reactivestreams:reactive-streams:1.0.3
....
This tells us that org.reactivestreams:reactive-streams-flow-adapters:1.0.2 was pulled in from com.softwaremill.sttp.client3:httpclient-backend_2.13:3.3.6.
If I can figure that out, how can I fix it? Is there a way other than just removing one of them?
In this case, reactive-streams-flow-adapters is subsumed by reactive-streams post-1.0.3, so I think we can exclude it from the dependency as follows:
libraryDependencies += ("com.softwaremill.sttp.client3" %% "httpclient-backend-zio" % "3.3.6")
.exclude("org.reactivestreams", "reactive-streams-flow-adapters"),
To ignore module-info.class we can add this to build.sbt:
ThisBuild / assemblyMergeStrategy := {
case "module-info.class" => MergeStrategy.discard
case x =>
val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
oldStrategy(x)
}
sbt:root> assembly
[info] Strategy 'discard' was applied to 10 files (Run the task at debug level to see details)
[success] Total time: 3 s, completed Jun 26, 2021 6:02:04 PM