I am creating a jdbc object in spark driver and I am using that in executor to access the db. So my concern is that is it the same connection object or executors would get a copy of connection object so there would be separate connection per partition?
Object
for a ConnectionPool
as per link below;
Executor
and shared by Cores
comprising that Executor
. I.e. not as a Singleton for all foreachPartition.
See https://medium.com/@ravishankar.nair/implementing-a-connectionpool-in-apache-sparks-foreachpartition-4add46dc8ae2. It's a good reference from medium.com.