I am trying to run a python package called mETL from PuTTY and use it in all the files contained in a folder. The python package is mETL and I am using it to load data contained in 3 .csv files called upload-A.csv, upload-B.csv and upload-C.csv
Everything works perfectly when I do this process manually using the following commands:
metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-A.csv config3.yml
metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-B.csv config3.yml
metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-C.csv config3.yml
All the data from every file is correctly uploaded or updated and the pickle files are updated accordingly.
But instead of doing this manually I want to make a loop that does this for all files contained in my 'folder_test/' folder, for which I tried the following Bash script:
folder_var=folder_test
for x in $folder_var
do
metl -m migration.pickle -t new_migration.pickle -s $x config3.yml
done
What happens after this is that the pickle files are created but no data is uploaded to the database.
Try this
for x in folder_test/*
do
metl -m migration.pickle -t new_migration.pickle -s "${x}" config3.yml
done