Why BroadcastExchange needs more driver memory?
When broadcasting, Spark can fail with the error org.apache.spark.sql.errors.QueryExecutionErrors#notEnoughMemoryToBuildAndBroadcastTableError (Spark 3.2.1):
Why BroadcastExchange needs more driver memory? If what broadcast does is to send data to all workers, why driver memory is a bottleneck?
Thanks.
Solution
Unfortunately executor side broadcast joins are not yet supported in Spark (see SPARK-17556). Currently all data of the broadcasted dataset is collected in the driver first to build an in-memory hash table which is then distributed to workers. This can result in high memory pressure on the driver.
- Spark "storage partitioned join" (SPJ)
- LEFT and RIGHT function in PySpark SQL
- I need to calculate profit/loss for given stock data set, ensuring that the first bought items are sold first
- Error when running a query involving ROUND function in spark sql
- How to add a constant column in a Spark DataFrame?
- Extracting several regex matches in PySpark
- How compute the percentile in PySpark dataframe for each key?
- Do stages in an application run parallel in spark?
- How to write spark streaming DF to Kafka topic
- What are the Spark transformations that causes a Shuffle?
- Spark Executor Fails to Connect to Driver in Cluster Standalone mode: "Connection refused: hostname/ip:randomport"
- How to handle an AnalysisException on Spark SQL?
- Write Delta format to Data Lake in AWS S3
- Apache Sedona on EMR version > 6.9.0: JavaPackage object is not callable
- Pyspark dataframe repartitioning puts all data in one partition
- Sort in descending order in PySpark
- Convert a Spark DataFrame to Pandas DF
- how to filter out a null value from spark dataframe
- Row object in RDD
- Filtering is supported only on partition keys of type string Hive
- Why does using getField() return a none type in AWS glue for scala for an Array of objects despite confirming it is there
- spark sql write.parquet overwrite issue
- Pyspark stream kafka debezium topic Error format, ETL
- Parallelism in AWS Glue
- Non Spark code (Plain Python) run on Spark cluster
- AnalysisException: mismatched input ';' expecting <EOF>
- How to just simple to insert data(rows) to delta table without strange merge syntax?
- Join Two 100k table taking longer than half hours
- Is there a difference between OUTER & FULL_OUTER in Spark SQL?
- How to escape the / while updating the columns using merge in spark.sql