I have a long list of filenames in filenames.txt file. These files are lzo compressed and I use lzop to decompress them for further processing in a pipeline.
cat filenames.txt | (xargs lzop -dc || true) | python lineprocessor.py > output.txt
So filenames are input to the lzop -dc line by line. Then they are decompressed and piped into the lineprocessor.py script that I have written. Finally the output of lineprocessor.py is written in output.txt.
The problem is that some files in filenames.txt are not properly compressed and lzop crashes and so does the whole pipeline. I added the || true to prevent this but it didn't help. lzop doesn't have the option to ignore the error. I don't care about the incorrectly compressed files.
Is there any way I can work around this problem easily? I want the pipeline to continue no matter what happens to lzop -dc command.
while read filename; do
lzop -fdc "$filename" | python lineprocessor.py
done < filenames.txt >> output.txt