bashloopsline

How do you run a command eg chmod, for each line of a file?


For example, right now I'm using the following to change a couple of files whose Unix paths I wrote to a file:

cat file.txt | while read in; do chmod 755 "$in"; done

Is there a more elegant, safer way?


Solution

  • Read a file line by line and execute commands: 4+ answers

    Because the main usage of ( and others shells like ) is to run other commands, there is not only 1 answer!!

        0. Shell command line expansion
        1. xargs dedicated tool
        2. while read with some remarks and consideration about parallel processing
        3. while read -u using dedicated fd, for interactive processing (sample)
        5. running with inline generated script

    Regarding the OP request: running chmod on all targets listed in file, xargs is the indicated tool. But for some other applications, small amount of files, etc...

    1. Read entire file as command line argument.

    If

    you could use shell command line expansion. Simply:

    chmod 755 $(<file.txt)
    

    This command is the simplier one.

    2. xargs is the right tool

    For

    For many binutils tools, like chown, chmod, rm, cp -t ...

    xargs chmod 755 <file.txt
    

    Could be used after a pipe on found files by find:

    find /some/path -type f -uid 1234 -print | xargs chmod 755
    

    If you have special chars and/or a lot of lines in file.txt.

    xargs -0 chmod 755 < <(tr \\n \\0 <file.txt)
    
    find /some/path -type f -uid 1234 -print0 | xargs -0 chmod 755
    

    If your command need to be run exactly 1 time for each entry:

    xargs -0 -n 1 chmod 755 < <(tr \\n \\0 <file.txt)
    

    This is not needed for this sample, as chmod accepts multiple files as arguments, but this matches the title of question.

    For some special cases, you could even define the location of the file argument in commands generated by xargs:

    xargs -0 -I '{}' -n 1 myWrapper -arg1 -file='{}' wrapCmd < <(tr \\n \\0 <file.txt)
    

    Test with seq 1 5 as input

    Try this:

    xargs -n 1 -I{} echo Blah {} blabla {}.. < <(seq 1 5)
    
    Blah 1 blabla 1..
    Blah 2 blabla 2..
    Blah 3 blabla 3..
    Blah 4 blabla 4..
    Blah 5 blabla 5..
    

    where your command is executed once per line.

    IMPORTANT PREAMBLE BEFORE CHAPTER 3.

    Doing loop under is generally a bad idea! There is a lot of warning about doing loop under shell!

    Before doing loop, think parallelisation and dedicated tools!!

    You could use for interact with and administrate dedicated tools. Some samples:

    3. while read and variants.

    For this, make sure to end the file with a newline character.

    As OP suggests,

    cat file.txt |
        while read in; do
            chmod 755 "$in"
        done
    

    will work, but there are 2 issues:

    So this could be better written:

    while read in; do
        chmod 755 "$in"
    done < file.txt
    

    But

    help read

    read: read [-r] ... [-d delim] ... [name ...]
        ...
        Reads a single line from the standard input... The line is split
        into fields as with word splitting, and the first word is assigned
        to the first NAME, the second word to the second NAME, and so on...
        Only the characters found in $IFS are recognized as word delimiters.
        ...
        Options:
          ...
          -d delim   continue until the first character of DELIM is read, 
                     rather than newline
          ...
          -r do not allow backslashes to escape any characters
        ...
        Exit Status:
        The return code is zero, unless end-of-file is encountered...
    

    In some cases, you may need to use

    while IFS= read -r in;do
        chmod 755 "$in"
    done <file.txt
    

    for avoiding problems with strange filenames. And maybe if you encounter problems with UTF-8:

    while LANG=C IFS= read -r in ; do
        chmod 755 "$in"
    done <file.txt
    

    While you use a redirection from standard inputfor reading file.txt`, your script cannot read other input interactively (you cannot use standard input for other input anymore).

    3.1 while read for limited number of concurrent parallel tasks

    If you plan to run a big number of repetitive tasks, using multiprocessing, you could do something like:

    maxProc=4
    ...
    wait4oneTask() {
        wait -np epid
        results[epid]=$?
        ...
        unset "running[$epid]"
    }
    ...
    while read file ;do
        ...
        exec {shC_Fd}>"${tmpLoc}_${file//\//_}"
        shellcheck -f gcc "$file" >&$shC_Fd 2>&1 &
        lpid=$!
        ...
        running[lpid]=''
        ((${#running[@]}>=maxProc)) && wait4oneTask
    done
    while ((${#running[@]})); do
        wait4oneTask
    done
    

    This is extracted from Parallel ShellCheck script: parShellCheck.sh a sample of parallelizing process and overall statistics of collected shellcheck remarks.

    4. while read, using dedicated fd.

    Syntax: while read ...;done <file.txt will redirect standard input to come from file.txt. That means you won't be able to deal with processes until they finish.

    This will let you use more than one input simultaneously, you could merge two files (like here: scriptReplay.sh), or maybe:

    You plan to create an interactive tool, you have to avoid use of standard input and use some alternative file descriptor.

    Constant file descriptors are:

    4.1 first

    You could see them by:

    ls -l /dev/fd/
    

    or

    ls -l /proc/$$/fd/
    

    From there, you have to choose unused numbers between 0 and 63 (more, in fact, depending on sysctl superuser tool) as your file descriptor.

    For this demo, I will use file descriptor 7:

    while read <&7 filename; do
        ans=
        while [ -z "$ans" ]; do
            read -p "Process file '$filename' (y/n)? " foo
            [ "$foo" ] && [ -z "${foo#[yn]}" ] && ans=$foo || echo '??'
        done
        if [ "$ans" = "y" ]; then
            echo Yes
            echo "Processing '$filename'."
        else
            echo No
        fi
    done 7<file.txt
    

    If you want to read your input file in more differents steps, you have to use:

    exec 7<file.txt      # Without spaces between `7` and `<`!
    # ls -l /dev/fd/
    
    read <&7 headLine
    while read <&7 filename; do
        case "$filename" in
            *'----' ) break ;;  # break loop when line end with four dashes.
        esac
        ....
    done
    
    read <&7 lastLine
    
    exec 7<&-            # This will close file descriptor 7.
    # ls -l /dev/fd/
    

    4.2 Same under

    Under , you could let him choose any free fd for you and store into a variable:
    exec {varname}</path/to/input:

    while read -ru ${fle} filename;do
        ans=
        while [ -z "$ans" ]; do
            read -rp "Process file '$filename' (y/n)? " -sn 1 foo
            [ "$foo" ] && [ -z "${foo/[yn]}" ] && ans=$foo || echo '??'
        done
        if [ "$ans" = "y" ]; then
            echo Yes
            echo "Processing '$filename'."
        else
            echo No
        fi
    done {fle}<file.txt
    

    Or

    exec {fle}<file.txt
    # ls -l /dev/fd/
    read -ru ${fle} headline
    
    while read -ru ${fle} filename;do
        [[ -n "$filename" ]] && [[ -z ${filename//*----} ]] && break
        ....
    done
    
    read -ru ${fle} lastLine
    
    exec {fle}<&-
    # ls -l /dev/fd/
    

    5. filtering input file for creating commands

    sed <file.txt 's/.*/chmod 755 "&"/' | sh
    

    This won't optimise forks, but this could be usefull for more complex (or conditional) operation:

    sed <file.txt 's/.*/if [ -e "&" ];then chmod 755 "&";fi/' | sh
    
    sed 's/.*/[ -f "&" ] \&\& echo "Processing: \\"&\\"" \&\& chmod 755 "&"/' \
        file.txt | sh
    

    This can be very useful if sed input is a feed instead of a file. Practical sample: Using rsync log output as sed input for deleting corresponding description file when a project file are deleted. See my answer to Remove file if a file with the same name but different extension doesn't exist in another directory which differ a lot from what SO asker did expect.