[SOLVED] aws s3 moving objects into glacier

aws s3 moving objects into glacier

Initial_State:

I have two S3 buckets termed as input_bucket and output_bucket. every second, my input_bucket filled with csv_files.

My Requirment: i want to move csv_files present in input_bucket(STANDARD_CLASS) to output_bucker(Glacier storage class)

Note: I want to move csv_files(40 days older from the date of creation)

please suggest some solution for this

Solution

There are multiple ways to achieve that:

1 Using replication (Easy and Recommended)

You don't need to write any code to achieve it. This solution will replicate files created in input_bucket to output_bucket and then delete them from input_bucket after 40 days.

Create replication rule between source (input_bucket) and source (output_bucket)
Add Expiration using S3 lifecycle configuration in the source bucket.

E.g.

<LifecycleConfiguration>
  <Rule>
    <ID>Transition Rule</ID>
    <Status>Enabled</Status>
    <Expiration>
      <Days>40</Days>
      <StorageClass>S3 Glacier</StorageClass>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

2 Using custom application (Not recommended)

Enable S3 triggers and then use Lambda function to create Dynamodb item with TTL enabled for 40 days. Dynamodb item should contain the S3 object path. You will also need to enable Dynamodb stream to trigger a Lambda function that would copy the object from input_bucket to output_bucket and then delete it from the source. This solution is expensive and not recommended.