javaapache-sparkcollectionshashmap

UnsupportedOperationException When Inserting into Map

I am using the Spark's collectAsMap function [Spark CollectAsMap to obtain a Map. In this map, when I do the put operation I am getting the following exception:

ERROR ApplicationMaster: User class threw exception: java.lang.UnsupportedOperationException
java.lang.UnsupportedOperationException
    at java.util.AbstractMap.put(AbstractMap.java:209)

Is the map obtained from collectAsMap unmodifiable?

Solution

  • I assume you are working in Java. You are getting this error because in java spark, collectAsMap returns a java wrapper around a scala map. In Spark 2.2, this wrapper is a custom class defined in this source file. As you can see, it does not define the put method, hence your error.

    A workaround could be to simply copy the map into a java HashMap as follows

    List<Tuple2<Integer, Integer>> list = new ArrayList<>();
list.add(new Tuple2<>(1,2));
list.add(new Tuple2<>(3,4));
Map<Integer, Integer> map = sc.parallelize(list)
    .mapToPair( x -> x )
    .collectAsMap();
    
Map<Integer, Integer> newMap = new HashMap<>(map);
newMap.put(7, 8);
System.out.println(newMap);

    That yields the expected {1=2, 3=4, 7=8}.