apache-sparkversionmahout

Mahout and Spark version compatibility


I am currently using Mahout 0.13.0. When I try to use spark-itemsimilarity I get an error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf

I suspect it may be an issue with Spark version, but I can't find what version of Spark do I need. I am currently using Spark 2.1.0.

So my question is, with which version of Spark is Mahout 0.13.0 compatible?


Solution

  • There are no binaries for Mahout 0.13.0 on Spark 2.1 / Scala 2.11, however it is possible to build Mahout for Spark 2.1 (unfortunately there are no profiles, have to do it all with D switches)

    to build you need to do the following:

    git clone http://github.com/apache/mahout

    cd mahout

    mvn clean package -Dspark.version=2.1.0 -Dscala.version=2.11.8 -Dscala.compat.version=2.11

    Also, Pat is correct- release coming hopefully very very soon, will have binaries.