sqlapache-sparkapache-spark-sqlsql-date-functions

Format string to datetime using Spark SQL

I am trying to convert and reformat a date column stored as a string using spark sql from something that looks like this...

30/03/20 02:00

to something that is a datetime column and looks like this...

2020-03-30 02:00 ('YYYY-MM-dd HH:mm')

I am not using python but simply writing sql in DBeaver directly to a spark datalake. Any help is greatly appreciated.

Solution

  • Use to_timestamp to parse string date into timestamp column and date_format to format it to desired pattern:

    select date_format(to_timestamp(sting_date, 'dd/MM/yy HH:mm'), 'yyyy-MM-dd HH:mm') as date