I have some scripts that I run which take a short while to run and produce a fair amount of output (e.g. yt-dlp, rsync). A lot of this output is unneeded so I want to be able to edit it (e.g. with sed). I also want this edited output to be displayed on the console while the script is still running (not just at the end), and I also want to capture this edited output into a variable.
I have been able to accomplish all of this except capturing the edited output into a variable. For example in the following command, I'm using yt-dlp to simulate downloading a playlist, and saving anything that looks like a URL, Warning or Error.
yt-dlp -s https://www.youtube.com/playlist?list=PL-cwa6ZvflaA7qryuX_m6Vyqt7ZwMi6xl 2>&1 \
| xargs -I % -n 1 echo % \
| sed -rn 's/.*((http.+)|^(WARNING|ERROR).+)/\1/gp'
This edits the output how I want and prints the output as the script runs, but if I use command substitution to capture this output then nothing gets printed.
I'm using Bash 5.2 and don't care about portability, if that helps.
=============================
EDIT:
In the linked question, there is no editing of the output in the solution, and if I try and add some editing then nothing gets printed until the loop terminates, e.g.:
exec 5>&1
VAR1=$(for i in {1..5}; do sleep 1; echo $i; done | sed s'/^/i=/' | tee >(cat - >&5))
or
exec 5>&1
VAR1=$(for i in {1..5}; do sleep 1; echo $i; done | xargs -I % -n 1 echo % | sed s'/^/i=/' | tee >(cat - >&5))
The only way I can get all of the behaviour that I want (editing the output of a program that runs for a while, and having this edited output printed to the console while that program is still running, and capturing the edited output in a variable) is to put the editing /within/ the loop, as follows:
exec 5>&1
VAR1=$(for i in {1..5}; do sleep 1; echo $i | sed s'/^/i=/'; done | tee >(cat - >&5))
However this doesn't address my original question since the loop in the linked question's answer is a stand-in for the long-running programs that I'm talking about (e.g. yt-dlp; rsync...), and there must be a way to accomplish this without having to compile my own versions of those programs.
Using /dev/stderr should achieve what was expected:
#!/usr/bin/env bash
VAR1=$(for i in {1..5}; do sleep 1; echo $i; done |
tee /dev/stderr |
sed s'/^/i=/')
echo 'Content of $VAR1 ->'
echo "$VAR1"
If you want edited output on the console:
#!/usr/bin/env bash
VAR1=$(for i in {1..5}; do sleep 1; echo $i; done |
stdbuf -oL sed 's/^/i=/' |
tee /dev/stderr)
echo 'Content of VAR1 ->'
echo "$VAR1"