apache-sparkapache-spark-sqlspark-java

Creating empty Spark dataframe and adding arbitrary values

I have created an empty dataframe and then trying to add columns and values to the dataframe.

Dataset<Row> runMetadata = sparkSession.emptyDataFrame();
runMetadata = runMetadata.withColumn("record_count", lit(count));
runMetadata = 
         runMetadata.withColumn("start_time",lit(currTimestamp));
runMetadata.show();

count is long and currTimestamp is a timestamp.

But 'runMetadata.show()' is only showing the column names and no values. Also i am trying to save this dataframe in bigquery, there also no value is getting appended, also i am not seeing any errors in logs.

Solution

  • When you use withColumn, you're defining a transformation that it's going to be applied to all the rows of your DataFrame. Your DataFrame has no rows, so there's no rows apply the new columns to.

    If you want a DataFrame with just one row you can create it using the createDataFrame method from the SparkSession passing a list with only one element.