pythonbashfor-loopall-files

Bash script to execute a python package on all files in a folder


I am trying to run a python package called mETL from PuTTY and use it in all the files contained in a folder. The python package is mETL and I am using it to load data contained in 3 .csv files called upload-A.csv, upload-B.csv and upload-C.csv

Everything works perfectly when I do this process manually using the following commands:

metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-A.csv config3.yml
metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-B.csv config3.yml
metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-C.csv config3.yml

All the data from every file is correctly uploaded or updated and the pickle files are updated accordingly.

But instead of doing this manually I want to make a loop that does this for all files contained in my 'folder_test/' folder, for which I tried the following Bash script:

folder_var=folder_test
for x in $folder_var
do
metl -m migration.pickle -t new_migration.pickle -s $x config3.yml
done

What happens after this is that the pickle files are created but no data is uploaded to the database.


Solution

  • Try this

    for x in folder_test/*
    do
    metl -m migration.pickle -t new_migration.pickle -s "${x}" config3.yml
    done