linuxshellamazon-s3aws-clilogrotate

logrotate: move only the new compressed files to S3


I have an application generating output and error logs. I need to compress them and keep 5 logs on server. Also, the logs should be copied to S3 bucket right after there are compressed.

lastaction seems to me the right place where I should write my script, since I want the files to be compressed.

My configuration file looks like this:

/var/log/nodejs/*out.log /var/log/nodejs/*err.log {
  size 10M
  missingok
  notifempty
  rotate 5 
  sharedscripts
  compress
  copytruncate
  dateext
  dateformat -%Y%m%d-%s
  olddir /var/log/nodejs/rotated
  lastaction
    echo $@
    INSTANCE_ID="`wget -q -O - http://instance-data/latest/meta-data/instance-id`"
    HOSTNAME=`hostname`
    BUCKET="my-logs"
    REGION="us-west-2"
    read DAY MONTH YEAR <<< `date "+%d %m %Y"`  
    aws s3 sync /var/log/nodejs/rotated/ "s3://$BUCKET/${INSTANCE_ID}_${HOSTNAME}/$YEAR/$MONTH/$DAY/" --region $REGION
  endscript
}

The problem here with the aws s3 sync is that if there are old logs from yesterday (in the 5 logs that are kept in the rotated folder), it will upload them again to the new folder of today.

Is there a way to get in lastaction (or other script provided by logroate) ONLY the files that were rotated now, but in their new location, so I could use aws s3 copy instead of sync?

For example: I print the args at the lastaction, and got:

/var/log/nodejs/app-out.log /var/log/nodejs/app-err.log

While I would like to get the new location:

/var/log/nodejs/rotated/app-out.log-20190131-1548925261.gz /var/log/nodejs/rotated/app-err.log-20190131-1548925261.gz

Solution

  • I found a solution...

    I added --exclude --include to the aws s3 sync command. Now my script looks like:

    /var/log/nodejs/*out.log /var/log/nodejs/*err.log {
      size 10M
      missingok
      notifempty
      rotate 5 
      sharedscripts
      compress
      copytruncate
      dateext
      dateformat -%Y%m%d-%s
      olddir /var/log/nodejs/rotated
      lastaction
        echo $@
        INSTANCE_ID="`wget -q -O - http://instance-data/latest/meta-data/instance-id`"
        HOSTNAME=`hostname`
        BUCKET="my-logs"
        REGION="us-west-2"
        read DAY MONTH YEAR <<< `date "+%d %m %Y"` 
        FORMAT=`date "+%Y%m%d"` 
        aws s3 sync /var/log/nodejs/rotated/ "s3://$BUCKET/${INSTANCE_ID}_${HOSTNAME}/$YEAR/$MONTH/$DAY/" --region $REGION --exclude "*" --include "*.log-$FORMAT*"
      endscript
    }