I am using aws-cli/1.22.97 from Linux and trying to download a single file from S3 bucket that contains objects (files) of storage class GLACIER and STANDARD.
As expected, I am able to download files of storage class STANDARD using aws s3 cp s3://bucket-name/pre_1/pre_2/file_name.ext . but when I try to download single file of storage class GLACIER it's failing.
I tried downloading single file of storage class GLACIER using aws s3 cp s3://bucket-name/pre_1/pre_2/file_name.ext . --storage-class STANDARD --force-glacier-transfer and it results in following error:
download failed: s3://bucket-name/pre_1/pre_2/file_name.ext to ./file_name.ext An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class
I also tried other variants of same command but get errors as below:
aws s3 cp s3://bucket-name/pre_1/pre_2/file_name.ext . --storage-class GLACIER --force-glacier-transfer
download failed: s3://bucket-name/pre_1/pre_2/file_name.ext to ./file_name.ext An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class
aws s3 cp s3://bucket-name/pre_1/pre_2/file_name.ext . --storage-class GLACIER
warning: Skipping file s3://bucket-name/pre_1/pre_2/file_name.ext. Object is of storage class GLACIER. Unable to perform download operations on GLACIER objects. You must restore the object to be able to perform the operation. See aws s3 download help for additional parameter options to ignore or force these transfers.
aws s3 cp s3://bucket-name/pre_1/pre_2/file_name.ext . --storage-class STANDARD
warning: Skipping file s3://bucket-name/pre_1/pre_2/file_name.ext. Object is of storage class GLACIER. Unable to perform download operations on GLACIER objects. You must restore the object to be able to perform the operation. See aws s3 download help for additional parameter options to ignore or force these transfers.
I read some posts here like this on SO that state for single file download, this format of cp command should work but for me it is not working. I don't know if the GLACIER is of type "Flexible Retrieval" or "Deep Archive" storage class.
Is there any way to download a single file of storage class GLACIER using AWS CLI without having to run two commands?
I am looking for a single command that I can run from CLI.
Is there any way to download a single file of storage class GLACIER using AWS CLI without having to run two commands?
No, there is no way to do this using AWS CLI.
Glacier is a special storage class which is not immediately available in the usual sense of the word.
AWS doesn't publicly disclose the tecnhology behind it, but people have been speculating that it's tape drives, Blu-ray discs or magnetic drives that just sit on a rack in a warehouse, not being connected to anything. The popular belief holds it that Glacier restores require physically moving data media around, using manual labor. Amazon is a company that is known for being able to perform this kind of operations efficiently. That's the (most probable) reason Glacier data is dirt cheap to store but expensive to download.
To copy something from Glacier, you'll need to wait for it to become available on S3 proper (i.e. restored from whatever medium they are sitting on while cold). It can take hours.
It would not be technically impossible to implement this flow as a single command, but the way AWS API calls are usually made is opening an HTTPS stream, sending a request and waiting for a response. They are designed for something that takes seconds and not hours.
This flow is much better implemented using asynchronous events or polling, as was alluded to in the comments.