javaapache-sparkspark-graphx

Making a Graph in Spark GraphX using Java bindings: What are those "evidence" arguments?


In the Spark GraphX javadoc the fromEdges() method has undocumented arguments like evidence$17. Presumably these are artifacts of the Scala implementation, but what should I do with them in Java?

public static <VD,ED> Graph<VD,ED> fromEdges(RDD<Edge<ED>> edges,
     VD defaultValue,
     StorageLevel edgeStorageLevel,
     StorageLevel vertexStorageLevel,
     scala.reflect.ClassTag<VD> evidence$16,
     scala.reflect.ClassTag<ED> evidence$17)

UPDATE: found some other Scala/Java examples, and it seems like the right thing is ClassTag$.MODULE$.apply(myClass) where myClass is the class of the VD or ED type, respectively. Not that this actually works, as it leads to later, mysterious exceptions, like java.lang.ArrayStoreException: java.lang.Integer deep in org.apache.spark.graphx.impl.EdgePartitionBuilder.toEdgePartition

UPDATE: Indeed, the ClassTag$.MODULE$.apply(Long.class) works for me. The error I was getting was due to passing in 1 for the default value instead of 1L.


Solution

  • The evidence arguments are filled in using ClassTag$.MODULE$.apply(myClass) where myClass is the class of the VD or ED type, respectively.

    If you use Long.class for the ED and VD types, make sure to use a matching value for the defaults when you call Graph.fromEdges(). In particular for Long.class use e.g. 1L for the default value instead of 1.