memoryapache-sparkpysparkhadoop-yarn

How does spark.python.worker.memory relate to spark.executor.memory?


This diagram is quite clear on the relationship between the different YARN and Spark memory related settings, except when it comes to spark.python.worker.memory.

How does spark.python.worker.memory fit into this memory model?

Are the Python processes governed by spark.executor.memory or yarn.nodemanager.resource.memory-mb?

Update

This question explains what the setting does, but doesn't answer the question concerning the memory governance, or how it relates to other memory settings.


Solution

  • Found this thread from the Apache-spark mailing list, and it appears that spark.python.worker.memory is a subset of the memory from spark.executor.memory.

    From the thread: "spark.python.worker.memory is used for Python worker in executor"