apache-sparkpysparkwarnings

Meaning of Apache Spark warning "Calling spill() on RowBasedKeyValueBatch"

I'm running a pyspark 2.2.0 job using the Apache Spark local mode and see the following warning:

WARN RowBasedKeyValueBatch: Calling spill() on RowBasedKeyValueBatch. Will not spill but return 0.

What could be the reason for this warning? Is this something I should care about or can I safely ignore it?

Solution

  • As indicated here this warning means that your RAM is full and that part of the RAM contents are moved to disk.

    See also the Spark FAQ

    Does my data need to fit in memory to use Spark?

    No. Spark's operators spill data to disk if it does not fit in memory, allowing it to run well on any sized data. Likewise, cached datasets that do not fit in memory are either spilled to disk or recomputed on the fly when needed, as determined by the RDD's storage level.