bashsedescapingbackslashsingle-quotes

how to eliminate trailing newlines and store the result in variables

As the title says, I want to eliminate trailing newlines and store the result in variables (Bash).

The following does not work.

[xxx@ ~]$ var=`echo 'abc\ndef\n' | sed 's/\\n$//g'`
[xxx@ ~]$ echo $var
abc\ndef\n

The following works!

[xxx@ ~]$ var=`echo 'abc\ndef\n' | sed 's/\\\\n$//g'`
[xxx@ ~]$ echo $var
abc\ndef

I thought that only one backslash escape was needed for sed since it is enclosed in single quotes, but I guess my understanding is wrong.

Please tell me why the second command works correctly?

Solution

  • Regarding:

    Please tell me why the second command works correctly?

    This command produces the shown output:

    $ echo 'abc\ndef\n' | sed 's/\\\\n$//g'
abc\ndef\n

    As for why you get this unexpected output with old, deprecated backticks:

    $ var=`echo 'abc\ndef\n' | sed 's/\\\\n$//g'`
$ echo "$var"
abc\ndef

    which is different from this expected output with modern command substitution ($(...)):

    $ var=$(echo 'abc\ndef\n' | sed 's/\\\\n$//g')
$ echo "$var"
abc\ndef\n

    it's because backslashes (\) inside backticks are handled in a non-obvious manner as stated and demonstrated at the top of the related FAQ, https://mywiki.wooledge.org/BashFAQ/082, and that is one of the reasons to use $(...) instead of backticks to execute commands.

    So, if you want to remove a trailing \n string using a pipe to sed then do this:

    $ var=$(echo 'abc\ndef\n' | sed 's/\\n$//g')
$ echo "$var"
abc\ndef

    otherwise, if you really want to remove newlines as your Subject line says then read on:

    echo 'abc\ndef\n' produces this output:

    $ echo 'abc\ndef\n'
abc\ndef\n

    The only newline present in that output is the one at the end of the line that makes that a valid POSIX text file. The \ns are literally the 2 characters \ and \n.

    You should use printf instead of echo for portability and consistency. In this case it would convert \n strings to newlines:

    $ printf 'abc\ndef\n'
abc
def

    Now, to get RID of the newlines - sed by default reads input 1 line at a time so you can't remove a newline from a string that sed sees as there cannot be a newline within 1 line that's part of newline-separated input. You can do things with sed to make it read multiple lines but it's more robust and portable to use awk instead, e.g.:

    to remove just the final newline:

    $ printf 'abc\ndef\n' | awk '{printf "%s%s", sep, $0; sep=ORS}'
abc
def             <--- no newline at the end

    or to remove all newlines:

    $ printf 'abc\ndef\n' | awk -v ORS= '1'
abcdef          <--- no newline at the end

    or to remove all except the final newline:

    $ printf 'abc\ndef\n' | awk -v ORS= '1; END{print RS}'
abcdef          <--- newline at the end

    or similar depending on whether you want the string to end in a newline (and so be a valid POSIX text line/file) or not.

    To execute a command, store it's result in a variable, and then print the contents of that variable, one way using command substitution would be:

    $ var=$(printf 'abc\ndef\n' | awk -v ORS= '1; END{print RS}')
$ printf '%s\n' "$var"
abcdef

    Note the use of $(...) instead of the long deprecated backticks (https://mywiki.wooledge.org/BashFAQ/082), and the necessary quotes "..." around the shell variable (see https://mywiki.wooledge.org/Quotes). Get into the habit of copy/pasting your code into http://shellcheck.net and it'll help you find and fix errors.