bashshellsedsubstitution

How to replace a large substring in a file?


Have a file containing multiple lines. But in some lines, there's a large substring need to be sorted. Now I got the substring and sorted substring, both stored as variables. Is there a way to replace the large substring?

i have tried sed -i "s?$subStr?$sortSub?" file but sed raised argument list too long error. didn't find relevant solutions. Is there a way to fix this?


Solution

  • Passing long strings as arguments to commands can be problematic due to length limits imposed by the OS. You seem to be encountering one of these limits. On my machine, I can only have 131071 characters in a single argument:

    $ sed "$(tr -c '#' '#' </dev/zero | head -c $(($(getconf ARG_MAX)/16-1)))" /dev/null
    $ sed "$(tr -c '#' '#' </dev/zero | head -c $(($(getconf ARG_MAX)/16)))"   /dev/null
    bash: /usr/bin/sed: Argument list too long
    $ echo $(($(getconf ARG_MAX)/16))
    131072
    

    Since you are using bash, the substitution can be done using the // form of parameter expansion, and the printf builtin which does not have the argument length limit.

    If your bash supports mapfile:

    mapfile -d '' data <file
    builtin printf '%s' "${data//"$subStr"/"$sortSub"}" >file
    

    Otherwise:

    data=$(cat file; echo .)
    data=${data%.}
    builtin printf '%s' "${data//"$subStr"/"$sortSub"}" >file
    

    Note that this will replace all occurrences of $subStr inside $data.