linuxbashpipestderr

Piped stderr but mongodump stdout lines appear in error log file

I have tried the below two options in crontab and expecting to get only stderr in the log file. However, some non-error lines related to mongodump command in the script file appear in the log file. I have seen various posts on stderr piping here, but couldn't get what I wanted. Where am I going wrong?

The two crontab options that are tried:

0 0 * * * /root/scripts/dev-vps_backup.sh >> logfile.log 2>&1 > /dev/null

0 0 * * * /root/scripts/dev-vps_backup.sh 2>&1 >> all.log | tee -a all.log err.log >&2

the bash script file:

#!/bin/bash

set -e

#start time var
start_time=$(TZ=Asia/Calcutta date +%d-%m-%Y_%H-%M-%S)
echo "Started Script: " $(basename "$0") "@ $start_time"

#########UPDATE THIS###########
### setting up variables here
######################## END of UPDATE

#create backup folders
mkdir -p $local_backupfolder
mkdir -p $local_dumps_backupfolder
mkdir -p $local_assets_backupfolder

# remote backup storage
remote_base_backupfolder=root@$remote_IP:/root/r-backups/$vps_name


# mongodump
gzfile=$local_dumps_backupfolder/$vps_name--dump--$start_time.gz
dumpMsg="$vps_name mongodump StartTime:$start_time, EndTime:"

if mongodump --gzip --archive=$gzfile ; then   ### THIS generates lines that are shown in stderr file
  echo "DONE: $dumpMsg $(TZ=Asia/Calcutta date +%d-%m-%Y_%H-%M-%S)"
else
  echo "ERR: $dumpMsg $(TZ=Asia/Calcutta date +%d-%m-%Y_%H-%M-%S)" | mailx -s "ERR: $vps_name mongodump" $recipient_email
  exit
fi

zipMsg1="$vps_name - $app_name1 - ZIP file for folder: $local_assets_folder1. StartTime:$start_time, EndTime:"

if zip -r $local_assets_backupfolder/$vps_name--$app_name1--assets-$(TZ=Asia/Calcutta date +%d-%m-%Y_%H-%M-%S).zip $local_assets_folder1 ; then
   echo "DONE: $zipMsg1 $(TZ=Asia/Calcutta date +%d-%m-%Y_%H-%M-%S)"
else
   echo "ERR: $zipMsg1 $(TZ=Asia/Calcutta date +%d-%m-%Y_%H-%M-%S)" | mailx -s "ERR: $vps_name - $app_name1 - assets local ZIP" $recipient_email
   exit
fi

some1-some_command  # to generate error

## some more lines

err.log file:

2024-08-02T10:33:01.116+0200    writing admin.system.version to archive '/root/backups/dumps/fvc1-vps--dump--02-08-2024_14-03-01.gz'
2024-08-02T10:33:01.118+0200    done dumping admin.system.version (1 document)
2024-08-02T10:33:01.118+0200    writing myca.syslogs to archive '/root/backups/dumps/fvc1-vps--dump--02-08-2024_14-03-01.gz'
.....some more lines related to mongodump command....
/root/scripts/fvc1-vps_backup.sh: line 61: some1-some_command: command not found

all.log file has the additional lines related to echo command and zip command.

EXPECTED: Only the last line in err.log file - which is the error. How do I achieve this?

UPDATE

tried this crontab:

59 11 * * * /root/scripts/fvc1-vps_backup.sh >> /root/scripts/logfile.log

resultant logfile:

Started Script:  fvc1-vps_backup.sh @ 02-08-2024_15-29-01
DONE: fvc1-vps mongodump StartTime:02-08-2024_15-29-01, EndTime: 02-08-2024_15-29-01
  adding: root/jobs/assets/ (stored 0%)
  adding: root/jobs/assets/tempfiles/ (stored 0%)
        ...... some more lines resulting from zip command.....
  adding: root/jobs/assets/invoiceDocs/ (stored 0%)
DONE: fvc1-vps - jobs - ZIP file for folder: /root/jobs/assets. StartTime:02-08-2024_15-29-01, EndTime: 02-08-2024_15-29-19

The log file doesnot include mongodump related lines. That means these lines are going to stderr (as show in the first log file result). Not sure why?

Solution

  • The second set of redirections appear suitable for what you want:

    shellscript 2>&1 >> all.log | tee -a all.log err.log >&2

    However it relies on output of all individual commands inside shellscript going where you want them (stdout or stderr).

    It appears that mongodump writes to stderr. If you want its output to appear in all.log but not err.log, you need to redirect its stderr to stdout. For example:

    if mongodump --gzip --archive=$gzfile 2>&1 ; then