[SOLVED] Spark JDBC read ends up in one partition only

Spark JDBC read ends up in one partition only

I have the below code snippet for reading data from a Postgresql table from where I am pulling all available data i.e. select * from table_name :

 jdbcDF = spark.read \
    .format("jdbc") \
    .option("url", self.var_dict['jdbc_url']) \
    .option("dbtable", "({0}) as subq".format(query)) \
    .option("user", self.var_dict['db_user']) \
    .option("password", self.var_dict['db_password']) \
    .option("driver", self.var_dict['db_driver']) \
    .option("numPartitions", 10) \
    .option("fetchsize", 10000) \
    .load()

Where var_dict is a dictionary containing my variables likes spark context , database creds etc.

Even when I am pulling millions of rows the result from below code returns 1 always:

partitions_num = jdbcDF.rdd.getNumPartitions()

Can someone advise if I am doing something wrong here? Ideally I would be expected to use maximum available resources rather then pulling the data to my master node only.

partitionColumn, lowerBound, upperBound cannot be used as my partition column is a timestamp and not numeric.

Solution

From spark 2.4.0, date and timestamp column is also supported for partitioning, https://issues.apache.org/jira/browse/SPARK-22814