rcsvappendwrite.table

How to append new rows into existing csv files where columns are in different order


I have 4 csv files and every week I get new raw data. I want to append the new data into the existing 4 csv file reports I have in a repository. the problem is that when I use either write.table or write.csv with the parameter "append = TRUE" the columns aren't matched therefore my rows are displaced.

for (report in names(clean_reports)) {
  data <- clean_reports[[report]]
  write.table(data,
              file = paste(report,".csv",sep = ""),
              append = TRUE,
              col.names = FALSE,
              row.names = FALSE,
              sep = ",")
}

I was expecting the columns appended into the correct position based on column name. Apparently write.table doesn't read the existing columns. Not sure what other workaround there is


Solution

  • When using write.table or write.csv with the append = TRUE parameter, the columns are not automatically matched based on column names. Instead, the new data is simply appended to the existing file without considering the column structure.

    To achieve the desired result of appending new data while ensuring the columns match, you can follow these steps:

    Read the existing CSV files into separate data frames. Append the new data to the corresponding data frames. Write the updated data frames back to the CSV files.

    # List of file names
    file_names <- c("report1.csv", "report2.csv", "report3.csv", "report4.csv")
    
    # Read existing CSV files into data frames
    existing_data <- lapply(file_names, read.csv)
    
    # Loop through the reports and append new data
    for (i in seq_along(file_names)) {
      report <- file_names[i]
      data <- clean_reports[[report]]  # Assuming clean_reports is a list with new data
      
      # Append new data to existing data frame
      if (exists(report, envir = .GlobalEnv)) {
        existing_data[[i]] <- rbind(existing_data[[i]], data)
      } else {
        existing_data[[i]] <- data
      }
      
      # Write the updated data frame back to the CSV file
      write.csv(existing_data[[i]], file = report, row.names = FALSE)
    }
    

    In this code, the existing CSV files are read into separate data frames using read.csv. Then, for each report, the new data is appended to the corresponding existing data frame using rbind. If the report data frame doesn't exist yet, it is created with the new data. Finally, the updated data frame is written back to the CSV file using write.csv, overwriting the existing file.