bashshellunixtar

Compress all files in all subfolders matching a pattern

In my root folder, there are many subfolders named data_x, data_easy, data_real, ... Each of them has a different hierarchy of subfolders. I need to compress files in ALL subfolders that match a pattern, in my case *test*.npy (all numpy files with 'test' in their name).

Currently, I do

shopt -s globstar; tar -czvf data.tar.gz data*/**/*test*.npy

But it is VERY slow. Sometimes it is faster to just compress ALL files without matching the pattern with **/*test*.npy. Is there a more efficient way to do it?

Solution

  • Try this to see if there is performance improvement :

    find data* -type f -name "*test*.npy" -print0 | tar -czvf data.tar.gz --null -T -