[SOLVED] Pre-Signed URL with boto3 and django-storages incorrect filepath

Pre-Signed URL with boto3 and django-storages incorrect filepath

For a file upload feature, I'm trying to set a FileField in Django using a presigned url but when the field is saved it seems that the aws bucket name is being added into the url so when I try to access the file from the django admin I get a Key does not exist error.

For the upload I first generate a presigned url on the backend:

try:
    s3_client = boto3.client(
        "s3",
        region_name=settings.AWS_S3_REGION_NAME,
        config=Config(signature_version="s3v4"),
        aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
        aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
    )

    s3_response = s3_client.generate_presigned_url(
        "put_object",
        HttpMethod="PUT",
        ExpiresIn=settings.AWS_SIGNED_URL_DURATION,
        Params={
            "Bucket": settings.AWS_STORAGE_BUCKET_NAME,
            "Key": f"{user.uuid}/{instance.uuid}/{payload.dict().get('filename')}",
            "ContentType": "application/octet-stream",
        },
    )
except ClientError as e:
    logger.exception(e)
    return 500, {"message": "Something went wrong."}

This is then used by the frontend client to upload the file - this all works ok and uploads a file to the following url:

https://<BUCKET_NAME>.s3.<BUCKET_REGION>.amazonaws.com/<USER_UUID>/<INSTANCE_UUID>/<FILE_NAME>

(I have checked in s3 and the file is there.)

The frontend then sends a patch request to update the DB with the url once the upload has been successful. The data looks like this:

{"file_field": "<USER_UUID>/<INSTANCE_UUID>/<FILE_NAME>"}

However, when I try to access the file from the django admin (or when I retrieve the instance data from my REST API), the file field shows a different url:

https://<BUCKET_NAME>.s3.<BUCKET_REGION>.amazonaws.com/<BUCKET_NAME>/<USER_UUID>/<INSTANCE_UUID>/<FILE_NAME>

(Notice the extra BUCKET_NAME after amazonaws.com/.

I can't see where this extra BUCKET_NAME is coming from - I feel like I'm missing something obvious here! Any help would be much appreciated, thanks!

EDIT:

Here is how I'm saving the FileField in an update view (using django-ninja):

if video:
    instance.video.name = video
    instance.save(update_fields=["video", "updated_at"])

where the video variable is a string containing the path of the file in s3: <USER_UUID>/<INSTANCE_UUID>/<FILE_NAME>

the field itself is just a simple FileField:

class MyClass(models.Model):
    video = models.FileField(blank=True, max_length=512)

Solution

As per this discussion on github, setting AWS_S3_ADDRESSING_STYLE to virtual and AWS_S3_ENDPOINT_URL to https://s3.amazonaws.com will prepend the bucket name to the url and not include it in the url path.

Hey, pretty sure this has to do with the AWS_S3_ADDRESSING_STYLE variable. Setting this variable to virtual, and setting your AWS_S3_ENDPOINT_URL to https://s3.amazonaws.com will automatically prepend your bucket name to the endpoint and not add it to the URL path.