arraysstringbashcsv

Converting a Bash array into a delimited string


I would like to know the following;

  1. Why the given non-working example doesn't work.
  2. If there are any other cleaner methods than those given in working example.

Non-working example

> ids=(1 2 3 4);echo ${ids[*]// /|}
1 2 3 4
> ids=(1 2 3 4);echo ${${ids[*]}// /|}
-bash: ${${ids[*]}// /|}: bad substitution
> ids=(1 2 3 4);echo ${"${ids[*]}"// /|}
-bash: ${"${ids[*]}"// /|}: bad substitution

Working example

> ids=(1 2 3 4);id="${ids[@]}";echo ${id// /|}
1|2|3|4
> ids=(1 2 3 4); lst=$( IFS='|'; echo "${ids[*]}" ); echo $lst
1|2|3|4

In context, the delimited string to be used in a sed command for further parsing.


Solution

  • Array to strings in bash

    1. Array to string, by using $IFS

    Because parentheses are used to delimit an array, not a string:

    ids="1 2 3 4";echo ${ids// /|}
    1|2|3|4
    

    Some samples: Populating $ids with two strings: a b and c d

    ids=("a b" "c d" e\ f)
    
    echo ${ids[*]// /|}
    a|b c|d e|f
    
    IFS='|';echo "${ids[*]}";IFS=$' \t\n'
    a b|c d|e f
    

    ... and finally:

    IFS='|';echo "${ids[*]// /|}";IFS=$' \t\n'
    a|b|c|d|e|f
    

    Where array is assembled, separated by 1st char of $IFS, but with space replaced by | in each element of array.

    When you do:

    id="${ids[@]}"
    

    you transfer the string build from the merging of the array ids by a space to a new variable of type string.

    Note: when "${ids[@]}" give a space-separated string, "${ids[*]}" (with a star * instead of the at sign @) will render a string separated by the first character of $IFS.

    what man bash says:

    man -Len -Pcol\ -b bash | sed -ne '/^ *IFS /{N;N;p;q}'
       IFS    The  Internal  Field  Separator  that  is used for word splitting
              after expansion and to split  lines  into  words  with  the  read
              builtin command.  The default value is ``<space><tab><newline>''.
    

    Playing with $IFS:

    printf "%q\n" "$IFS"
    $' \t\n'
    

    Literally a space, a tabulation and (meaning or) a line-feed. So, while the first character is a space. the use of * will do the same as @.

    But:

    {
        IFS=: read -a array < <(echo root:x:0:0:root:/root:/bin/bash)
        
        echo 1 "${array[@]}"
        echo 2 "${array[*]}"
        OIFS="$IFS" IFS=:
        echo 3 "${array[@]}"
        echo 4 "${array[*]}"
        IFS="$OIFS"
    }
    1 root x 0 0 root /root /bin/bash
    2 root x 0 0 root /root /bin/bash
    3 root x 0 0 root /root /bin/bash
    4 root:x:0:0:root:/root:/bin/bash
    

    Note: The line IFS=: read -a array < <(...) will use : as separator, without setting $IFS permanently. This is because output line #2 present spaces as separators.

    1.1 Using function, localize $IFS

    To just print array

    printArry() {
        local IFS="$1"
        shift
        echo "$*"
    }        
    
    printArry @ "${ids[@]}"
    a b@c d@e f
    

    Or to merge array in place.

    mergeArry() {
        local IFS="$1"
        local -n _array_to_merge=$2
        _array_to_merge=("${_array_to_merge[*]}")
    }
    
    declare -p ids
    declare -a ids=([0]="a b" [1]="c d" [2]="e f")
    
    mergeArry '#' ids
    declare -p ids
    declare -a ids=([0]="a b#c d#e f")
    

    2. Array of strings to array of strings ([@] vs [*])

    There is a notable difference between:

    Read carefully: man '-Pless +/Special\ Parameters' bash

    For this, I will quote each argument in order to not be splitted by $IFS at command line expansion, using double-quotes to permit variable expansion.

    ids=('a b c' 'd e f' 'g h i')
    
    printf '<< %s >>\n' "${ids[@]// /|}"
    
    << a|b|c >>
    << d|e|f >>
    << g|h|i >>
    
    printf '<< %s >>\n' "${ids[*]// /|}"
    
    << a|b|c d|e|f g|h|i >>
    

    Where:

    ( IFS='@'; printf '<< %s >>\n' "${ids[*]// /|}" )
    << a|b|c@d|e|f@g|h|i >>
    

    Note: ${var// /something} will replace every spaces by something, but ${var[*]} will merge array by using only one 1st character:

    ( IFS='FOO'; printf '<< %s >>\n' "${ids[*]// / BAR }" )
    << a BAR b BAR cFd BAR e BAR fFg BAR h BAR i >>
    

    And yes: by using ${var// / ... }, you could replace 1 space by anthing you want, including more spaces.

    3. Array to string, by using printf

    As we see, using $IFS is limited to only 1 character. If you need to use more characters to be inserted between your fields. You have to use printf:

    ids=("a b" "c d" e\ f)
    sep=" long separator "
    printf -v string "%s$sep" "${ids[@]}"
    echo "${string%$sep}"
    a b long separator c d long separator e f
    

    Note: this syntax work but is something limited, see further!

    3.1 Array to string, by using printf, into a function

    In order to support special characters as % or * in separator, the function have to prevents

    printArry() {
        local sep=$1 string
        shift
        printf -v string "%s${sep//%/%%}" "$@"
        echo "${string%"$sep"}"
    }        
    
    printArry ' long separator ' "${ids[@]}"
    a b long separator c d long separator e f
    
    printArry '*' "${ids[@]}"
    a b*c d*e f
    
    printArry '%' "${ids[@]}"
    a b%c d%e f
    

    Or to merge array in place.

    mergeArry() {
        local sep=$1 string
        local -n _array_to_merge=$2
        printf -v string "%s${sep//%/%%}" "${_array_to_merge[@]}"
        _array_to_merge=("${string%"$sep"}")
    }
    
    ids=("a b" "c d" e\ f)
    mergeArry ' another separator ' ids
    declare -p ids
    declare -a ids=([0]="a b another separator c d another separator e f")
    
    ids=("a b" "c d" e\ f)
    mergeArry '*' ids
    declare -p ids
    declare -a ids=([0]="a b*c d*e f")
    

    3.2 Array to string, but submitting array as argument:

    Instead of using nameref for array variable, you could use:

    MergeWithSep() {
        if [[ $1 == -v ]]; then local outvar=$2 sep=$3 string; shift 3
        else local outvar sep=$1 string; shift
        fi
        printf -v string "%s${sep//%/%%}" "$@"
        if [[ -n $outvar ]]; then
            printf -v $outvar "%s" "${string%"$sep"}"
        else
            echo "${string%"$sep"}"
        fi
    }
    
    ids=("a b" "c d" e\ f)
    MergeWithSep ' || ' "${ids[@]}"
    a b || c d || e f
    
    MergeWithSep $'\n - ' " - "Hello\ world. 'This is a sentence.'
     - Hello world.
     - This is a sentence.
    
    MergeWithSep -v var ', '  {A..Z}
    echo $var.
    A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z.
    

    4. Merge array to string using bash's parameter expansion

    Yet another way TMTOWTDI: But as for working, we have to empty $IFS, I prefer to use this in a function for localize $IFS.

    printArry () { 
        local -n _array_to_print=$2
        local IFS=
        local _string_to_print="${_array_to_print[*]/#/"$1"}"
        echo "${_string_to_print/#"$1"}"
    }
    

    Note you could replace # by % as

    mergeArry () { 
        local -n _array_to_merge=$2
        local IFS=
        _array_to_merge=("${_array_to_merge[*]/#/"$1"}")
        _array_to_merge=("${_array_to_merge/#"$1"}")
    }
    

    4.1 Little variant, based on length of separator:

    printArry () { 
        local -n _array_to_print=$2
        local IFS=
        local _string_to_print="${_array_to_print[*]/#/"$1"}"
        echo "${_string_to_print:${#1}}"
    }
    
    mergeArry () { 
        local -n _array_to_merge=$2
        local IFS=
        _array_to_merge=("${_array_to_merge[*]/#/"$1"}")
        _array_to_merge=("${_array_to_merge:${#1}}")
    }
    

    Or

    printArry () { 
        local -n _array_to_print=$2
        local IFS=
        local _string_to_print="${_array_to_print[*]/%/"$1"}"
        echo "${_string_to_print::-${#1}}"
    }
    

    4.1 Then

    MergeWithSep() {
        if [[ $1 == -v ]]; then local outvar=$2 sep=$3 string; shift 3
        else local outvar sep=$1 string; shift
        fi
        local IFS=
        string=${@/#/"$sep"}
        if [[ -n $outvar ]]; then
            printf -v outvar %s "${string/#"$sep"}"
        else
            echo "${string/#"$sep"}"
        fi
    }
    

    5. Comparison

    To do...

    See also