I have an application generating output and error logs. I need to compress them and keep 5 logs on server. Also, the logs should be copied to S3 bucket right after there are compressed.
lastaction seems to me the right place where I should write my script, since I want the files to be compressed.
My configuration file looks like this:
/var/log/nodejs/*out.log /var/log/nodejs/*err.log {
size 10M
missingok
notifempty
rotate 5
sharedscripts
compress
copytruncate
dateext
dateformat -%Y%m%d-%s
olddir /var/log/nodejs/rotated
lastaction
echo $@
INSTANCE_ID="`wget -q -O - http://instance-data/latest/meta-data/instance-id`"
HOSTNAME=`hostname`
BUCKET="my-logs"
REGION="us-west-2"
read DAY MONTH YEAR <<< `date "+%d %m %Y"`
aws s3 sync /var/log/nodejs/rotated/ "s3://$BUCKET/${INSTANCE_ID}_${HOSTNAME}/$YEAR/$MONTH/$DAY/" --region $REGION
endscript
}
The problem here with the aws s3 sync is that if there are old logs from yesterday (in the 5 logs that are kept in the rotated folder), it will upload them again to the new folder of today.
Is there a way to get in lastaction (or other script provided by logroate) ONLY the files that were rotated now, but in their new location, so I could use aws s3 copy instead of sync?
For example: I print the args at the lastaction, and got:
/var/log/nodejs/app-out.log /var/log/nodejs/app-err.log
While I would like to get the new location:
/var/log/nodejs/rotated/app-out.log-20190131-1548925261.gz /var/log/nodejs/rotated/app-err.log-20190131-1548925261.gz
I found a solution...
I added --exclude --include to the aws s3 sync command. Now my script looks like:
/var/log/nodejs/*out.log /var/log/nodejs/*err.log {
size 10M
missingok
notifempty
rotate 5
sharedscripts
compress
copytruncate
dateext
dateformat -%Y%m%d-%s
olddir /var/log/nodejs/rotated
lastaction
echo $@
INSTANCE_ID="`wget -q -O - http://instance-data/latest/meta-data/instance-id`"
HOSTNAME=`hostname`
BUCKET="my-logs"
REGION="us-west-2"
read DAY MONTH YEAR <<< `date "+%d %m %Y"`
FORMAT=`date "+%Y%m%d"`
aws s3 sync /var/log/nodejs/rotated/ "s3://$BUCKET/${INSTANCE_ID}_${HOSTNAME}/$YEAR/$MONTH/$DAY/" --region $REGION --exclude "*" --include "*.log-$FORMAT*"
endscript
}