hadoop-yarnemr

Executors on CORE node in EMR cluster


In the EMR cluster I have 1 MASTER, 1 CORE and 4 TASK nodes. It seems the YARN is creating CPU bound SPARK executors not only on TASK nodes, but also on CORE node. Is there any option to prevent it (like 'yarn.app.mapreduce.am.labels' does to instruct YARN to create APP MANAGER only on MASTER or CORE).


Solution

  • AM runs inside a container. At most, you can specify which node to run your AMs on.

    These refs should help you: