bashinotifyinotifywait

Are delete events considered close_write in inotifywait?


I have a simple inotifywait script that watches for FTP file uploads to be closed and then moving them to a aws s3. It seems to be working except that in the inotify logs, it indicates that the file was not found ( although the file was indeed uploaded to s3 ). The s3 move command moves the file to the cloud and deletes it locally. Could this be because inotifywait detects deleting the file as a close_write event ? Why is inotify seems to be executing the commands twice ?

TARGET=/home/*/ftp/files

inotifywait -m -r -e close_write $TARGET |
    while read directory action file
    do
        if [[ "$file" =~ .*mp4$ ]]
        then
            echo COPY PATH IS "$directory$file"
            aws s3 mv "$directory$file" s3://bucket
        fi
    done

example logs:

Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
COPY PATH IS /home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4
COPY PATH IS /home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4
COPY PATH IS /home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4
move: ../user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4 to s3://bucket/user-cam-1_00_20220516114055.mp4
upload: ../user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4 to s3://bucket/user-cam-1_00_20220516114055.mp4
move failed: ../user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4 to s3://bucket/user-cam-1_00_20220516114055.mp4 [Errno 2] No such file or directory: '/home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4'
rm: cannot remove '/home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4': No such file or directory
                                                                                                                  

Solution

  • Cleaned-up your script and added some safety with quotes and check for already processed file in case the filesystem triggers duplicate events for same file.

    #!/usr/bin/env bash
    
    # Prevents expanding pattern without matches
    shopt -s nullglob
    
    # Expands pattern into an array
    target=(/home/*/ftp/files/)
    
    # Creates temporary directory and cleanup trap
    declare -- tmpdir=
    if tmpdir=$(mktemp -d); then
      trap 'rm -fr -- "$tmpdir"' EXIT INT
    else
      # or exit error if it fails
      exit 1
    fi
    
    # In case no target matches, exit error
    [ "${#target[@]}" -gt 0 ] || exit 1
    
    s3move() {
      local -- p=$1
      local -- tmp="$tmpdir/$p"
      printf 'Copy path is: %s\n' "$p"
      # Moves the file to temporary dir
      # so it is away from inotify watch dir ASAP
      mv -- "$p" "$tmp"
    
      # Then perform the slow remote copy to s3 bucket
      # Remove the echo onces it is ok
      echo aws s3 mv "$p" s3://bucket
      
      # File has been copied to s3, tmp file no longer needed
      rm -f -- "$tmp"
    }
    
    while read -r -d '' p; do
      # Skip if file does not exist, as it has already been moved away
      # case of a duplicate event for already processed file
      [ -e "$p" ] || continue
      s3move "$p"
    done < <(
      # Good practice to spell long option names in a script
      # --format will print null-delimited full file path
      inotifywait \
        --monitor \
        --recursive \
        --event close_write \
        --includei '.*\.mp4$' \
        --format '%w%f%0' \
        "${target[@]}" 2>/dev/null
    )