amazon-web-servicesamazon-s3backupbackup-strategies

Automatically delete old backups from S3 and move monthly to glacier


I have set up Gitlab to save a daily backup to an Amazon S3 bucket. I want to keep a monthly backup one year back on glacier and daily backups one week back on standard storage. Is this cleanup strategy viable and doable using S3 lifecycle rules? If yes, how?


Solution

  • Amazon S3 Object Lifecycle Management can Transition storage classes and/or Delete (expire) objects.

    It can also work with Versioning such that different rules can apply to the 'current' version and 'all previous' versions. For example, the current version could be kept accessible while pervious versions could be transitioned to Glacier and eventually deleted.

    However, it does have the concept of a "monthly backup" or "weekly backup". Rather, rules are applied to all objects equally.

    To achieve your monthly/weekly objective, you could:

    Or, you could use the same Lifecycle rules on all backups but write some code that deletes unwanted backups at various intervals (eg every day deletes a week-old backup unless it is the first backup of the month). This code would be triggered as a daily Lambda function.