I am using a shell expansion pattern (or is it called globstar?) with rsync (it is performing much faster for me than filtering, when the input directory contains 10's of thousands of files). But I am having trouble achieving my desired output-directory structure. See the self contained example below to see what I mean:
mkdir -p targetvolume/targetdir/a/my_folder
mkdir -p targetvolume/targetdir/b/my_folder
mkdir -p targetvolume/targetdir/c/my_folder
mkdir outputdir
touch targetvolume/targetdir/a/my_folder/file1.txt
touch targetvolume/targetdir/b/my_folder/file2.txt
touch targetvolume/targetdir/c/my_folder/file3.txt
Now if I run this:
sync -ahv targetvolume/targetdir/**/my_folder/ ./outputdir
the outputdir will look like this
#ls outputdir
file1.txt file2.txt file3.txt
But I want it to look like this:
a/my_folder/file1.txt
b/my_folder/file2.txt
c/my_folder/file3.txt
I can't figure this out. adding the -R flag doesn't work for me because it includes the full absolute path to the files (which is very cumbersome when working with my actual data/directory, because the outputdir/directories/becomes/super/nested).
-R does seem to be the option you need. It's just that when given the full path, the program won't know where it should be cut, which exact part you want to replicate at the destination. (Remember, it's the shell that expands the ** wildcard, while rsync just gets a list of filenames and has no idea how they came to be.) The man page gives two options:
Either, put an extra ./ in the path to mark the cut point. Indeed this seems to do what you want:
rsync -ahv -R targetvolume/targetdir/./**/my_folder/ ./outputdir
producing outputdir/a/my_folder/file1.txt etc.
Or, cd deeper into the tree first. Though this'll require adjusting the destination too:
dest=$PWD/outputdir
cd targetvolume/targetdir/
rsync -ahv -R ./**/my_folder/ "$dest"