apache-sparkpyspark

Converting Double type column to date format type pyspark retuning

I have a dataframe with double column (20180310), i want to convert it to Date format (2018-03-10).

I used this function:

df.withColumn(
    "new_col", 
    F.to_date(F.col("old_col").cast("string"), "yyyyMMdd")
)

but it's returning error message or Null.

Do you have any idea please?

Solution

  • Because when you cast from double to string, the column will have this form: 2.018031E7

    cast to integer before casting to string:

    df.withColumn( "new_col", F.to_date(F.col("old_col").cast("integer").cast("string"), "yyyyMMdd"))