javascalaapache-sparkjava-8eclipse-luna

Compilation error while using Tuple2 in Spark Java Application


I am trying out Spark Programming examples using Java 1.8 in Eclipse Luna and have the following code -

JavaPairRDD<String, Integer> counts = ones
                .reduceByKey(new Function2<Integer, Integer, Integer>() {
                    @Override
                    public Integer call(Integer i1, Integer i2) {
                        return i1 + i2;
                    }
                });
List<Tuple2<String, Integer>> output = counts.collect(); //Compilation Error

I am using M2Eclipse to build and create the jar and using spark-submit to execute the jar in my local. The jar is working and printing the correct output but Eclipse always shows the above mentioned line as a compilation error - The type Tuple2 is not generic; it cannot be parameterized with arguments <String, Integer>

Even the programming examples referred in the Spark webpage uses the same notation for Tuple2. https://spark.apache.org/docs/0.9.0/java-programming-guide.html

I am not able to understand why Eclipse is showing it as a compilation error since the return type of the collect call is a List<Tuple2<String,Integer>>

Any help is greatly appreciated.


Solution

  • As mentioned by @Holger in the comments, 2 scala-library jars were added to the build path. Removed the earlier version and compilation errors disappeared.