[SOLVED] Extract files with lftp and iterate over the extracted files

Extract files with lftp and iterate over the extracted files

I am using lftp to extract some CSV data files from a FTP-server, with the following command:

lftp -e 'set ftp:use-mdtm false; set ftp:timezone Europe/Berlin; mirror --newer-than=now-1days --no-recursion --verbose -i "INERGIA.*\.csv" / /mnt/trailstone/itpf/DataInput; bye' -u [USERNAME],[PASSWORD] [SERVER-NAME]

This runs perfectly. However, after extracting the CSV files, I want to iterate over each of them in order to modify some of the data in the files with a Python script. Is there a way via lftp to get a hold of the extracted files directly by for instance assigning them to a variable?

At this moment I just do the following:

cd "/mnt/trailstone/itpf/DataInput/"

for f in $(find *.csv -ctime -1) do 
   python /mnt/trailstone/itpf/OnlineDataProcessing/OnlineExtraDataDownloader/changeDelimiter.py $f
done

But I would prefer getting a hold of the files directly through lftp.

Solution

I'm supposing you want to do this because find might be hitting files with last status changed accidentally amongst previously fetched cvs files.

I'm not too familiar with lftp but I can think of a workaround: have lftp download the files to a temporary location, do the python stuff and subsequently move the modified files to their final destination. Alternatively extract the downloaded file list lftp's standard output (piped trough awk and grep, or whatever suits you best), populate a variable with that and then loop with that instead of $(find *.csv -ctime -1).