[SOLVED] Issues adding spark-redis connector as a dependency

Issues adding spark-redis connector as a dependency

I am trying to use the spark-redis connector provided by RedisLabs. I could try it in my local environment, downloading the necessary jars files.

But when I try to execute it in EMR, I need to add it in my built.sbt file, something like this:

libraryDependencies += "redis.clients" % "jedis" % "2.9.0"
libraryDependencies += "org.apache.commons" % "commons-pool2" % "2.5.0"
libraryDependencies += "RedisLabs" % "spark-redis" % "0.3.2"

But I am getting an error saying that can not resolve that dependency. It can not found the spark-redis library.

Is there any other way to do it? Any suggestion?

Solution

I was able to compile adding spark-streaming to the dependencies and including the dependecy for spark streaming from an external URL:

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.3.0" % "provided"

libraryDependencies += "redis.clients" % "jedis" % "2.9.0"
libraryDependencies += "org.apache.commons" % "commons-pool2" % "2.5.0"

libraryDependencies += "RedisLabs" % "spark-redis" % "0.3.2" from "https://dl.bintray.com/spark-packages/maven/RedisLabs/spark-redis/0.3.2/spark-redis-0.3.2.jar"