I am uploading files through Python scripts to DigitalOcean Spaces (AWS S3-Compatible Object Storage).
Uploading files is working well; the part I am not so sure about is how to properly access those files. I had to set
IMAGES_STORE_S3_ACL = 'public-read'
so I am able to display the image that I uploaded to the storage. Otherwise, I would see the Permission Denied error. The way I display the file is that I concatenate path to the file, something like file_path = f'{storage_domain}/my_files/{file_name}'.
When I put this path to the browser, it displays the file. But basically, anyone can concatenate this URL and display the files from the storage.
When uploading the files to the storage, there were these outputs in the console:
Making request for OperationModel(name=PutObject) with params: {'url_path': '/path/to/file.jpg', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-meta-width': '749', 'x-amz-meta-height': '562', 'x-amz-acl': 'public-read', 'Cache-Control': 'max-age=172800', 'Content-Type': 'image/jpeg', 'User-Agent': 'Botocore/1.34.83 ua/2.0 os/macos#23.4.0 md/arch#arm64 lang/python#3.11.8 md/pyimpl#CPython cfg/retry-mode#legacy', 'Content-MD5': 'something here==', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x10778fbf0>, 'auth_path': '/path/to/file.jpg', 'url': 'https://location.digitaloceanspaces.com/path/to/file.jpg', 'context': {'client_region': 'region', 'client_config': <botocore.config.Config object at 0x107189210>, 'has_streaming_input': True, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'bucket_name', 'params': {'Bucket': 'bucket_name', 'Key': 'path/to/file.jpg', 'Body': <_io.BytesIO object at 0x10778fbf0>, 'Metadata': {'width': '749', 'height': '562'}, 'ACL': 'public-read', 'CacheControl': 'max-age=172800', 'ContentType': 'image/jpeg'}}, 'input_params': {'Bucket': 'bucket_name', 'Key': 'path/to/file.jpg'}, 'signing': {'region': 'location', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'fra1'}]}}}
Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1071891d0>>
Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x106070fe0>
Event before-sign.s3.PutObject: calling handler <function remove_arn_from_signing_path at 0x106073560>
Event before-sign.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x1071a8c10>>
Calculating signature using v4 auth.
CanonicalRequest:
PUT
/path/to/file.jpg
cache-control:max-age=172800
content-md5:something==
content-type:image/jpeg
host:location.digitaloceanspaces.com
x-amz-acl:public-read
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20240412T204214Z
x-amz-meta-height:562
x-amz-meta-width:749
cache-control;content-md5;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-meta-height;x-amz-meta-width
UNSIGNED-PAYLOAD
StringToSign:
AWS4-HMAC-SHA256
20240412T204214Z
20240412/location/s3/aws4_request
something_here
Signature:
something_here
This output is coming out of the boto3 module (I assume) - I am not sure if I can access this data from my Python script.
However, regarding the proper and safe way of accessing the uploaded files - I think my way of concatenating the URL to the file is not very safe. What's the right way to do it?
(My answer is based on my knowledge of Amazon S3, not specifically Digital Ocean.)
Any data uploaded to S3 (and therefore Digital Ocean) is private by default. That is, only authenticated users with necessary permissions can access the data.
If you would like anyone in the world to access the object, then you can use ACL=public-read as you mention, or you can create a Bucket Policy that makes the whole bucket (or a subset of the bucket) public. This will make the object (or bucket) publicly accessible to anyone.
If you only want access to be available to users defined in Digital Ocean, then requests to access an object would need to be done via authenticated API calls (eg using the AWS CLI or boto3, together with Digital Ocean credentials).
For example, if you want to download an object to your own computer, you would have DO credentials saved to a configuration file and the you could use aws s3 cp s3://bucketname/foo.txt . to download the object. It will work because you have credentials that grant access to the object.
Alternatively, let's say you have a web application and you would like to display a private image on a web page, or perhaps provide a link to a private object. Since the object is private, you could use Amazon S3 pre-signed URLs, which provide time-limited access to private objects in Amazon S3. The app would generate a link that includes authorisation information and therefore permits access to that object within a defined period of time. (Basically, the URL has an additional signature that is a security hash that grants permission to access the private object.)