bashrsync

rsync : Recursively sync all files while ignoring the directory structure


I am trying to create a bash script for syncing music from my desktop to a mobile device. The desktop is the source.

Is there a way to make rsync recursively sync files but ignore the directory structure? If a file was deleted from the desktop, I want it to be deleted on the device as well.

The directory structure on my desktop is something like this.

    Artist1/
        Artist1/art1_track1.mp3
        Artist1/art1_track2.mp3
        Artist1/art1_track3.mp3
    Artist2/
        Artist2/art2_track1.mp3
        Artist2/art2_track2.mp3
        Artist2/art2_track3.mp3
    ...

The directory structure that I want on the device is:

    Music/
        art1_track1.mp3
        art1_track2.mp3
        art1_track3.mp3
        art2_track1.mp3
        art2_track2.mp3
        art2_track3.mp3
    ...

Solution

  • 1. If numbers of subdirs are fixed:

    Simply:

    rsync -a --delete --include=*.mp3 --exclude=* \
        pathToSongs/Theme*/Artist*/. destuser@desthost:Music/.
    

    would do the job if you're path hierarchy has a fixed number of level.

    WARNING: if two song file do have exactly same name, while on same destination directory, your backup will miss one of them!

    2. Without find, by using bash's globstar

    If else, and for answering strictly to your ask ignoring the directory structure you could use 's shopt -s globstar feature:

    shopt -s globstar
    rsync -a --delete --include=*.mp3 --exclude=* \
        pathToSongsRoot/**/. destuser@desthost:Music/.
    

    At all, there is no need to fork to find command.

    From man bash:

    globstar
           If  set,  the  pattern ** used in a pathname expansion context will
           match all files and zero or more  directories  and  subdirectories.
           If the pattern is followed by a /, only directories and subdirecto‐
           ries match.
    

    2.1 Recursively sync all files while ignoring the directory structure

    For answering strictly to question, there must no be limited to an extension:

    shopt -s globstar
    rsync -d --delete sourceRoot/**/. destuser@desthost:destRoot/.
    

    With this, directories will be copied too, but without content. All files and directories would be stored on same level at destRoot/.

    WARNING: If some different files with same name exists in defferents directories, they would simply be overwrited on destination, durring rsync, for finaly storing randomly only one.

    3. By using find

    When you use find, you may have to ensure special character in filenames to be correctly handled. For this you should use -print0 option to separate filenames by null chars (0x0), then tell rsync to read stdin by using null chars as separator:

    find first/source/path second/source/path -type f -name \*.mp3 -print0 |
        rsync -av0 --files-from - --no-relative . destuser@desthost:destRoot/.
    

    (Note the little 0 after -av, mean -a -v -0.)

    This syntax is based on relative path! If you plan to use full path you will use full path as find arguments and replace dot (current path) by a slash (root) in rsync command:

    find /first/source/path /second/source/path -type f -name \*.mp3 -print0 |
        rsync -av0 --files-from - --no-relative / destuser@desthost:destRoot/.
    

    Note the use of -print0 in printf command and -0 in rsync command, which ensure correct handling of special file names!

    For more speeding up

    Think parallelization, see:

    Speed up rsync with Simultaneous/Concurrent File Transfers?
    3 - 4 tricks for speeding up rsync.