apache-sparkspark-jdbcapache-spark-sql-repartition

Apache Spark - passing jdbc connection object to executors


I am creating a jdbc object in spark driver and I am using that in executor to access the db. So my concern is that is it the same connection object or executors would get a copy of connection object so there would be separate connection per partition?


Solution

  • See https://medium.com/@ravishankar.nair/implementing-a-connectionpool-in-apache-sparks-foreachpartition-4add46dc8ae2. It's a good reference from medium.com.