python-3.xdataframepysparkavrospark-avro

How to append data in existing AVRO file using Python


I have a dataframe with similar schema, I need to append the data into the AVRO file. I don't like to add the avro file into folder as a part. For your information, my AVRO file is not into the folder as a part. Can you please help me to solve the task.


Solution

  • You can write the data by using mode overwrite while writing the dataframe.

    But the part file is created as spark is distributed processing and each executor spits out a files based on the amount of data