I am using Apache Spark version 1.2.1 and Scala version 2.10.4. I am trying to get the example MovieLensALS working. However, I am running into errors with scopt library which is a requirement in the code. Any help would be appreciated. My build.sbt is as follows:
name := "Movie Recommender System"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.1"
libraryDependencies += "org.apache.spark" %% "spark-graphx" % "1.2.1"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.2.1"
libraryDependencies += "com.github.scopt" %% "scopt" % "3.2.0"
resolvers += Resolver.sonatypeRepo("public")
and the errors I am getting are the following:
Exception in thread "main" java.lang.NoClassDefFoundError: scopt/OptionParser
at MovieLensALS.main(MovieLensALS.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scopt.OptionParser
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 8 more
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
On running sbt assembly to build the jar, I receive the following errors:
[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly
[error] assembly
[error] ^
Edit: As per Justin Piphony's suggestion the solution that was listed in sbt's GitHub page helped fix this error. Basically creating a file assembly.sbt in the project/ directory and adding the line
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
Note that the version should be added according to the version in use.
You need to package scopt
in your jar. sbt doesn't do this by default. To create this fat jar, you need to use sbt-assembly