hadoopapache-sparkh2osparkling-water

H2o Number of Executors not working

I start the sparkling-shell with the following command.

./bin/sparkling-shell --num-executors 4 --executor-memory 4g --master yarn-client

I only ever get two executors. Is this an H2o problem, YARN problem, or Spark problem?

Mike

Solution

  • I got over the problem by changing the following four values in cloudera manager 

    Setting                                  Value
yarn.scheduler.maximum-allocation-vcores 8  
yarn.nodemanager.resource.cpu-vcores     4 
yarn.nodemanager.resource.cpu-vcores     4 
yarn.scheduler.maximum-allocation-mb     16 GB