I'm trying to figure out a way to send files to S3 to a signed URL that has arbitrary "directory" and "filename". Understanding that S3 doesn't have directories.
I can successfully generate the signed url and upload an arbitrary file name using the below:
Lambda to create the signed url:
def lambda_handler(event, context):
def create_presigned_post(bucket_name, object_name="${filename}",
fields=None, conditions=None, expiration=3600):
"""Generate a presigned URL S3 POST request to upload a file
:param bucket_name: string
:param object_name: string
:param fields: Dictionary of prefilled form fields
:param conditions: List of conditions to include in the policy
:param expiration: Time in seconds for the presigned URL to remain valid
:return: Dictionary with the following keys:
url: URL to post to
fields: Dictionary of form fields and values to submit with the POST
:return: None if error.
"""
# Generate a presigned S3 POST URL
s3_client = boto3.client('s3')
try:
response = s3_client.generate_presigned_post(bucket_name,
object_name,
Fields=fields,
Conditions=conditions,
ExpiresIn=expiration)
except ClientError as e:
logging.error(e)
return None
# The response contains the presigned URL and required fields
return response
bucketName = "myBucket"
s3client = boto3.client("s3")
signed_url = create_presigned_post(bucketName)
return signed_url
Request to upload the file:
session = boto3.Session(profile_name='dev')
object_name = 'test2.txt'
bucket = 'myBucket'
response = {
"url": "https://mys3url.com",
"fields": {
"key": "${filename}",
"AWSAccessKeyId": "...",
"x-amz-security-token": "...",
"policy": "...",
"signature": "..."
}
}
if response is None:
exit(1)
# Demonstrate how another Python program can use the pre-signed URL to upload a file
with open(object_name, 'rb') as f:
s3_object_name = f"this/is/a/test/dir/{object_name}"
files = {'file': (s3_object_name, f)}
http_response = requests.post(response['url'], data=response['fields'], files=files)
# If successful, returns HTTP status code 204
print(f'File upload HTTP status code: {http_response.status_code}')
print(f'{http_response.content}')
This works mostly fine with file test2.txt being written to the buckets "root".
However i would like to upload it to this/is/a/test/dir/ or any other directory in the S3 bucket.
I do understand that pre-signed urls are meant to be restrictive in nature and that this restriction might be by design, however if s3 has no directory structure would simply naming the file make it write to a "directory"?
The use of ${filename} allows me to write any filename to S3 but seems to strip any directory structure from the filename.
Useful links that have helped get me this far.
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html
Following @Quassnoi's comment, I'm going to work around the limitation. There may be a way to solve this however i couldn't find it.
@AnonCoward: POST requests to S3 are a little bit special, because they are designed to work with multipart data sent from an HTML form. In a PUT request, you would have to presign the exact key, but the POST policy supports the placeholder for the filename that comes with the form. The RFC for multipart form data explicitly requires that the server disregard any path information that might be present with the filename (and real web browsers don't send it), but I'm not sure if S3 actually strips the path if you do send it.