scalaapache-sparkapache-spark-sqlspark-jdbc

Does df.write.jdbc handle JDBC pool connection?

Do you know if the following line can handle jdbc pool connection:

df.write
  .mode("append")
  .jdbc(url, table, prop)

Do you have any idea? Thanks

Solution

  • I don't think so.

    spark.read.jdbc requests Spark SQL to create a JDBCRelation. Eventually buildScan is executed that in turn calls JDBCRDD.scanTable that leads to JdbcUtils.createConnectionFactory(options) for JDBCRDD.

    With that, you see driver.connect(options.url, options.asConnectionProperties) and unless driver deals with connection pooling Spark SQL does not do it.

    (just noticed that you asked another question)

    df.write.jdbc is similar. It leads to JDBCRelation again that uses the same RDD.