[SOLVED] Shapeless error while trying trying to use PureConfig and Circe for scala

Shapeless error while trying trying to use PureConfig and Circe for scala

shapeless.DefaultSymbolicLabelling  
shapeless.DefaultSymbolicLabelling$.instance(shapeless.HList)

getting this error while using both pureconfig and circe. I'm using spark 3.1.2 with spark k8s operator.

Solution

This error is because of conflicting shapeless library versions. Spark 3.1.2 ships with shapeless 2.3.3 whereas both these packages need shapeless 2.3.7. To solve this I followed the steps mentioned here which involve shading ie renaming the dependency.

For SBT

If you are using the sbt-assembly plugin to create your JARs you can shade shapeless by adding to your assembly.sbt file the following setting:

assembly / assemblyShadeRules := Seq(ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll)

Maven

The maven-shade-plugin can shade shapeless by adding to your pom.xml file the following block:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.0.0</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <createDependencyReducedPom>false</createDependencyReducedPom>
        <relocations>
            <relocation>
                <pattern>shapeless</pattern>
                <shadedPattern>shapelesspureconfig</shadedPattern>
            </relocation>
        </relocations>
    </configuration>
</plugin>