linuxbashshellgrepcat

How is it possible that a command chain ending in `cat` prints nothing, but executing `cat again on the same file prints a line?

I run this command sequence:

grep -Fxv "sudo: myname : 1 incorrect password attempt ; TTY=pts/3 ; PWD=/home/myname/workspace ; USER=root ; COMMAND=/usr/bin/pip uninstall ansible" ~/.log-scanner/blacklist > temp && mv temp ~/.log-scanner/blacklist && cat ~/.log-scanner/blacklist

and the output is an empty line. However, if I then enter the following command in the command prompt:

cat ~/.log-scanner/blacklist

It outputs:

sudo: myname : 1 incorrect password attempt ; TTY=pts/3 ; PWD=/home/myname/workspace ; USER=root ; COMMAND=/usr/bin/pip uninstall ansible

which is the exact line that I wanted to exclude from the file.

How is this possible? I can't understand how two consecutive cat calls can have different results.

I expected the output to be the same; either both cases to output the line that I tried to exclude, either both cases to output nothing.

I tried interspersing the cat commands with sync, but the results remain the same.

Edit: For more context, what I'm trying to do is, go through the lines of a file, and delete each literal occurrence of that line in another file, like this:

while read -r line || [[ -n $line ]]
do
    grep -Fxv "$line" "$blacklist" > temp && mv temp "$blacklist"
    # sed -i "\#$line#d" "$blacklist"
done < "$whitelist"

but a line is leftover and I can't understand why. When I tried to debug it, I came across the problem that I described above.

Solution

  • How is it possible that a command chain ending in cat prints nothing, but executing cat again on the same file prints a line?

    One reasonably likely way is if the cat in the first list isn't executed, but the file designated in the second command exists and contains a line.

    In your particular example, the cat in the first list would not be executed if either the preceding grep command or the preceding mv command failed, on account of these list being formed with && operators. grep fails (in the sense of exiting with non-zero status) if it doesn't match any lines, which seems a non-trivial possibility in your example. That would not ordinarily provoke a diagnostic.

    Your particular grep command will also fail on errors, and the mv command might fail for a variety of reasons, and there could be failures at the shell level too, such as while setting up redirections. All of these alternatives would ordinarily yield diagnostic messages as well, however.

    I can't understand how two consecutive cat calls can have different results.

    Even if both commands were in fact executed, both reading a file of the same name, with no other commands between them in that shell session, it would still be possible for them to produce different output. Remember always that your system does many things concurrently, or at least pseudo-concurrently. You could get different output if some other process running in the background or in a different session modified the file.

    For more context, what I'm trying to do is, go through the lines of a file, and delete each literal occurrence of that line in another file

    Your code will have the issue you describe when the file being filtered is initially non-empty, but all its lines are blacklisted.

    Suggested fix

    mv the temp file over the original unless an error occurs in grep, which you can detect by a more discriminating examination of grep's exit status. Perhaps this is what you thought you were doing already. When the -q option is not given to grep, it distinguishes between failing to match any lines (status 1) and encountering any kind of error (status > 1). Thus:

    grep -Fxv "$line" "$blacklist" > temp
[ $? -lt 2 ] && mv temp "$blacklist"