bashgzipxargspv

gzip on multiple cores with pv progression bar

Following this 2010 question Gzip with all cores, I would like to gzip files using multiple core and indicate a progress bar with pv tool.

How do I improve this code?

CORES=$(grep -c '^processor' /proc/cpuinfo)
find /source -type f -print0 | xargs -0 -n 1 -P $CORES gzip -9

I would like to show remaining time and show the progression bars running in parallel.

Do you have any other best alternatives as of 2018?

Thanks.

Solution

  • Use GNU Parallel which has a progress bar or an eta:

    find ... -print0 | parallel -0 --progress gzip -9 {}

    Or

    find ... -print0 | parallel -0 --eta ...

    Or

    find ... -print0 | parallel -0 --bar ...