variablespysparkexprnullif

Using variable in expression with Nullif fails

I'm using the code below to convert an empty string to Null. I don't get an error but it remains an empty string.
I suspect an incorrect use of the variable col_name in expr

for col_name in ['col1', 'col2']:
    df_new = df \
        .withColumn(col_name, F.expr(f"nullif('{col_name}', '')"))
Solution

  • What you really want to do is this. Be aware that the for-loop will create and overwrite df_new, so the final column only be changed.

    from pyspark.sql import functions as f

df_new = df
for col_name in ['col1', 'col2']:
    df_new = df_new.withColumn(col_name, f.expr(f"nullif({col_name}, '')"))