How to add multiple empty columns to a PySpark Dataframe at specific locations

I tried researching for this a lot but I am unable to find a way to execute and add multiple columns to a PySpark Dataframe at specific positions.

I have the dataframe that looks like this:

Customer_id   First_Name   Last_Name

I want to add 3 empty columns at 3 different positions and my final resulting dataframe needs to look like this:

Customer_id Address First_Name Email_address Last_Name Phone_no

Is there an easy way around it, like the way you can do with reindex on python?

Solution

# Creating a DataFrame.
from pyspark.sql.functions import col, lit
df = sqlContext.createDataFrame(
    [('1','Moritz','Schulz'),('2','Sandra','Schröder')],
     ('Customer_id','First_Name','Last_Name')
)
df.show()
+-----------+----------+---------+
|Customer_id|First_Name|Last_Name|
+-----------+----------+---------+
|          1|    Moritz|   Schulz|
|          2|    Sandra| Schröder|
+-----------+----------+---------+

You can use lit() function to add empty columns and once created you can use SQL's select to reorder the columns in the order you wish.

df = df.withColumn('Address',lit(''))\
       .withColumn('Email_address',lit(''))\
       .withColumn('Phone_no',lit(''))\
       .select( 
           'Customer_id', 'Address', 'First_Name',
           'Email_address', 'Last_Name', 'Phone_no'
       )
df.show()
+-----------+-------+----------+-------------+---------+--------+
|Customer_id|Address|First_Name|Email_address|Last_Name|Phone_no|
+-----------+-------+----------+-------------+---------+--------+
|          1|       |    Moritz|             |   Schulz|        |
|          2|       |    Sandra|             | Schröder|        |
+-----------+-------+----------+-------------+---------+--------+

As suggested by user @Pault, a more concise & succinct way -

df = df.select(
    "Customer_id", lit('').alias("Address"), "First_Name",
    lit("").alias("Email_address"), "Last_Name", lit("").alias("Phone_no")
)
df.show()
+-----------+-------+----------+-------------+---------+--------+
|Customer_id|Address|First_Name|Email_address|Last_Name|Phone_no|
+-----------+-------+----------+-------------+---------+--------+
|          1|       |    Moritz|             |   Schulz|        |
|          2|       |    Sandra|             | Schröder|        |
+-----------+-------+----------+-------------+---------+--------+