pythonarraysdataframefile-organization

How to import multiple csv files at once


I have 30 csv files of wind speed data on my computer- each file represents data at a different location. I have written code to calculate the statistics I need to run for each site; however, I am currently pulling in each csv file individually to do so(see code below):

from google.colab import files
data_to_load = files.upload()

import io
df = pd.read_csv(io.BytesIO(data_to_load['Downtown.csv']))

Is there a way to pull in all 30 csv files at once so each file is run through my statistical analysis code block and spits out an array with the file name and the statistic calculated?


Solution

  • use a loop

    https://intellipaat.com/community/17913/import-multiple-csv-files-into-pandas-and-concatenate-into-one-dataframe

    import glob
    
    import pandas as pd
    
    # get data file names
    
    local_path = r'/my_files'
    
    filenames = glob.glob(local_path + "/*.csv")
    
    dfs = [pd.read_csv(filename)) for filename in filenames]
    
    
    # if needed concatenate all data into one DataFrame
    
    big_frame = pd.concat(dfs, ignore_index=True)
    

    Also you can try put data online: github or google drive and read from there https://towardsdatascience.com/3-ways-to-load-csv-files-into-colab-7c14fcbdcb92