Save command output to a file and to a variable at the same time

Ubuntu 22.04

I have long running command, which prints output to console.

I would like to redirect that command output (stdout and stderr) to a log file immediately while there is some output for the command, and at the same time I would like to have this same output saved to a variable as well. I do not want any rows printed in the console.

Basically I want this:

output=$(${long_running_command} 2>&1)
echo "${output} >> /tmp/file.log"

The output is saved to a variable, this is all good. Only problem with this code above is that output is printed to a log file only after the long_running_command has been finished.

I am looking a way to achieve that log file write happens immediately when the command produces some output.

I have tried this also:

output=$(${long_running_command} >> /tmp/file.log 2>&1)

For this I can get redirection to log file immediately while there is anything outputted. But here the $output variable is completely empty.

Let's say the long_running_command example is the following:

output=$(for i in 1 2 3 4 5; do echo "$(date): Sleep: $i"; sleep $i; done >> /tmp/file.log 2>&1)
echo "$output"

While the command is executing, and I tail the log file, I can get the output immediately, while it happens. But the issue is, the $output variable is completely empty.

# tail -100f /tmp/file.log
Wed Aug 14 10:40:20 PM EEST 2024: Sleep: 1
Wed Aug 14 10:40:21 PM EEST 2024: Sleep: 2
Wed Aug 14 10:40:23 PM EEST 2024: Sleep: 3
Wed Aug 14 10:40:26 PM EEST 2024: Sleep: 4
Wed Aug 14 10:40:30 PM EEST 2024: Sleep: 5

# output=$(for i in 1 2 3 4 5; do echo "$(date): Sleep: $i"; sleep $i; done >> /tmp/file.log 2>&1)
# echo "$output"

<--- nothing, empty row

This looks like a job for tee ...

If you wish to capture all output (stdout + stderr) in the variable:

$ output="$(for i in 1 2 3 4 5; do date; ./do_some_stuff; sleep 1; done 2>&1 | tee -a file.log )"

Where:

• ./do_some_stuff - does not exist so it should generate an error
• 2>&1 - redirect stderr to stdout
• tee -a file.log - split input into 2 streams; one stream is -appended to file.log, second stream goes to stdout

Results:

$ typeset -p output
declare -- output="Wed Aug 14 15:01:04 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:01:05 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:01:06 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:01:07 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:01:08 CDT 2024
bash: ./do_some_stuff: No such file or directory"

$ cat file.log
Wed Aug 14 15:01:04 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:01:05 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:01:06 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:01:07 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:01:08 CDT 2024
bash: ./do_some_stuff: No such file or directory

One approach if you wish stderr to only go to the file:

$ output="$(for i in 1 2 3 4 5; do date; ./do_some_stuff; sleep 1; done 2>>file.log | tee -a file.log )"

Where:

• 2>>file.log - explicitly append stderr to file.log so ...
• only stdout is piped to tee -a file.log

Results:

$ typeset -p output
declare -- output="Wed Aug 14 15:02:21 CDT 2024
Wed Aug 14 15:02:22 CDT 2024
Wed Aug 14 15:02:23 CDT 2024
Wed Aug 14 15:02:25 CDT 2024
Wed Aug 14 15:02:26 CDT 2024"

$ cat file.log
Wed Aug 14 15:02:21 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:02:22 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:02:23 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:02:25 CDT 2024
bash: ./do_some_stuff: No such file or directory
Wed Aug 14 15:02:26 CDT 2024
bash: ./do_some_stuff: No such file or directory