bashsedwhile-loop

Why does sed in bash remove leading and trailing whitespaces from a string?


This is a MRE of a larger bash script, that replaces characters in filenames based on a list of files and a character map. Instead of a file list from a file I'm reading in two "files" from the var $filelist

#!/usr/bin/env bash

declare -A char_map

char_map["1"]="-"
char_map["2"]="+"

filelist=$(echo -e "./mydir/ number1number2withspaceoutside \n./mydir/number1 number2 with space inside")

while IFS= read current_file
do
    [[ -z "$current_file" ]] && continue

    path=$(dirname "$current_file")
    filename=$(basename "$current_file")
    fixed_filename="$filename"

    for bad in "${!char_map[@]}"
    do
        echo "before sed: '$fixed_filename'"
        fixed_filename=$(echo $fixed_filename | sed s/$bad/${char_map[$bad]}/g)
        echo "after sed: '$fixed_filename'"
        echo "--------------------"
    done

    if [ "$fixed_filename" == "$filename" ]
    then
        echo "nothing bad found: $current_file"
        continue
    fi
    
    echo "moving '$current_file' to '$path/$fixed_filename'"
    echo "================================================================================================"

done < <(echo "$filelist")

Output:

before sed: ' number1number2withspaceoutside '
after sed: 'number1number+withspaceoutside'
--------------------
before sed: 'number1number+withspaceoutside'
after sed: 'number-number+withspaceoutside'
--------------------
moving './mydir/ number1number2withspaceoutside ' to './mydir/number-number+withspaceoutside'
================================================================================================
before sed: 'number1 number2 with space inside'
after sed: 'number1 number+ with space inside'
--------------------
before sed: 'number1 number+ with space inside'
after sed: 'number- number+ with space inside'
--------------------
moving './mydir/number1 number2 with space inside' to './mydir/number- number+ with space inside'
================================================================================================

A strange side effect of the sed command on line 21 is the removal of leading and trailing spaces in the filenames. The effect is visible on the second line of the output:

after sed: 'number1number+withspaceoutside'

The spaces are gone... This is, for once, a side effect I appreciate but I don't understand it.

I can't recreate it in my terminal.

echo " bla " | sed s/l/-/g

outputs

b-a or for visibility ' b-a '

Can somebody explain to me why the spaces get removed and how I can prevent it if needed? I guess the explanation will also clear up the reason why it does not do it in the terminal.

I assume it has something to do with how the filenames are read into the while loop, but I'm not certain.


Solution

  • echo $fixed_filename | sed s/$bad/${char_map[$bad]}/g
    

    $fixed_filename is unquoted, which means it's subject to word splitting in which spaces are collapsed. Quote it as "$fixed_filename" to preserve whitespace.

    ShellCheck diagnoses the problem and shows how to fix it:

    Line 21:
            fixed_filename=$(echo $fixed_filename | sed s/$bad/${char_map[$bad]}/g)
                             ^-- SC2001 (style): See if you can use ${variable//search/replace} instead.
                                  ^-- SC2086 (info): Double quote to prevent globbing and word splitting.
                                                          ^-- SC2086 (info): Double quote to prevent globbing and word splitting.
    >>                                                         ^-- SC2086 (info): Double quote to prevent globbing and word splitting.
    
    Did you mean:
            fixed_filename=$(echo "$fixed_filename" | sed s/"$bad"/"${char_map[$bad]}"/g)