I have a Django REST Framework application that has a model with an ImageField:
def upload_to(instance, filename):
return f'images/model1/{instance.owner.pk}/${filename}'
class Model1(TimeStampedModel):
owner = models.ForeignKey(User, on_delete=models.CASCADE)
image = models.ImageField(upload_to=upload_to, null=True, blank=True)
I am attempting to configure the storage backend to use S3 and localstack from my docker where localstack and the application both run in docker:
version: "3.9"
services:
app:
build:
context: .
dockerfile: ./Dockerfile
volumes:
- .:/commas-api
restart: always
command: sh -c "/app/start-server-local.sh"
environment:
STAGE: 'local'
AWS_ACCESS_KEY_ID: 'dummy'
AWS_SECRET_ACCESS_KEY: 'dummy'
AWS_STORAGE_BUCKET_NAME: 'MyBucket'
AWS_S3_ENDPOINT_URL: 'http://localstack:4566'
expose:
- 8000
ports:
- "8000:8000"
depends_on:
localstack:
condition: service_healthy
links:
- localstack
localstack:
image: localstack/localstack
ports:
- "4566:4566"
environment:
PROVIDER_OVERRIDE_S3: v2
volumes:
- ./localstack:/etc/localstack/init/ready.d
volumes:
commas-api:
where I have the following settings:
# AWS Storage config
AWS_S3_ENDPOINT_URL = os.environ.get('AWS_S3_ENDPOINT_URL')
AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
AWS_S3_REGION_NAME = os.environ.get('AWS_S3_REGION', 'us-east-1')
AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
AWS_QUERYSTRING_AUTH = False
# S3 Static data config
STATIC_LOCATION = 'static'
STATIC_URL = f'{AWS_S3_ENDPOINT_URL}/{STATIC_LOCATION}/'
STATICFILES_STORAGE = 'commas_api.storage_backends.StaticStorage'
# S3 Media data config
PUBLIC_MEDIA_LOCATION = 'media'
MEDIA_URL = f'{AWS_S3_ENDPOINT_URL}/{PUBLIC_MEDIA_LOCATION}/'
DEFAULT_FILE_STORAGE = 'commas_api.storage_backends.PublicMediaStorage'
STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")]
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
This will work to upload the files. So when I create a new instance of Model1 with an image, it will be created in the S3 bucket and the URL of the image will be something like:
http://localstack:4566/MyBucket/media/images/model1/2/example.jpg
However, this is not accessible through my web browser (or from my frontend) but instead exists at:
http://localhost:4566/MyBucket/media/images/model1/2/example.jpg
Notice the localhost instead of localstack.
What I am trying to determine is how do I upload images and the URL provided would be the correct URL that I could load in a browser and see the image (so the frontend can load it).
My thought process was to use AWS_S3_CUSTOM_DOMAIN instead (especially since I want to use a CDN in prod). In my attempt to do this, I changed the following environment variables in my app for docker-compose:
STAGE: 'local'
AWS_ACCESS_KEY_ID: 'dummy'
AWS_SECRET_ACCESS_KEY: 'dummy'
AWS_STORAGE_BUCKET_NAME: 'MyBucket'
AWS_S3_URL_PROTOCOL: 'http'
AWS_S3_CUSTOM_DOMAIN: 'MyBucket.s3.localhost.localstack.cloud:4566'
and settings to:
# AWS Storage config
AWS_S3_CUSTOM_DOMAIN = os.environ.get('AWS_S3_CUSTOM_DOMAIN')
AWS_S3_URL_PROTOCOL = os.environ.get('AWS_S3_URL_PROTOCOL', 'https')
AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
AWS_S3_REGION_NAME = os.environ.get('AWS_S3_REGION', 'us-east-1')
AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
AWS_QUERYSTRING_AUTH = False
# S3 Static data config
STATIC_LOCATION = 'static'
STATIC_URL = f'{AWS_S3_URL_PROTOCOL}://{AWS_S3_CUSTOM_DOMAIN}/{STATIC_LOCATION}/'
STATICFILES_STORAGE = 'commas_api.storage_backends.StaticStorage'
# S3 Media data config
PUBLIC_MEDIA_LOCATION = 'media'
MEDIA_URL = f'{AWS_S3_URL_PROTOCOL}://{AWS_S3_CUSTOM_DOMAIN}/{PUBLIC_MEDIA_LOCATION}/'
DEFAULT_FILE_STORAGE = 'commas_api.storage_backends.PublicMediaStorage'
STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")]
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
And confirmed that my bucket is accessible on my browser at http://MyBucket.s3.localhost.localstack.cloud:4566
but then boto3 is unable to authenticate into the bucket when I run the application and I get the following stack trace when I attempt to upload any images to the app (The example here is when I collectstatic when the app starts):
app-1 | Traceback (most recent call last):
app-1 | File "/app/manage.py", line 21, in <module>
app-1 | main()
app-1 | File "/app/manage.py", line 17, in main
app-1 | execute_from_command_line(sys.argv)
app-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
app-1 | utility.execute()
app-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute
app-1 | self.fetch_command(subcommand).run_from_argv(self.argv)
app-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv
app-1 | self.execute(*args, **cmd_options)
app-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute
app-1 | output = self.handle(*args, **options)
app-1 | File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 209, in handle
app-1 | collected = self.collect()
app-1 | File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 135, in collect
app-1 | handler(path, prefixed_path, storage)
app-1 | File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 368, in copy_file
app-1 | if not self.delete_file(path, prefixed_path, source_storage):
app-1 | File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 278, in delete_file
app-1 | if self.storage.exists(prefixed_path):
app-1 | File "/usr/local/lib/python3.9/site-packages/storages/backends/s3boto3.py", line 463, in exists
app-1 | self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)
app-1 | File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call
app-1 | return self._make_api_call(operation_name, kwargs)
app-1 | File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 960, in _make_api_call
app-1 | raise error_class(parsed_response, operation_name)
app-1 | botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
In my top setup, the bucket is accessible by the application so I know my bucket permissions and CORS are alright. Has anyone dealt with this before and know how to setup django_storages with an S3 client and localstack to the host localhost.localstack.cloud so the image URL of the created Django model is accessible from my browser?
Any help would be greatly appreciated! Thank you.
I encountered the same challenge. When using Django with LocalStack, Django requires the address of LocalStack, while the frontend accesses the LocalStack container via localhost. This situation is primarily relevant in my development environment. Here's how I addressed it:
class CustomStorage(S3Boto3Storage):
"""Custom Storage for XY files."""
bucket_name = settings.AWS_XYFILES_BUCKET_NAME
if settings.DEBUG:
custom_domain = '{0}/{1}'.format(
settings.AWS_S3_CUSTOM_DOMAIN, settings.AWS_XYFILES_BUCKET_NAME
)
You'll need the following environment variables:
AWS_S3_CUSTOM_DOMAIN=localhost:4566
AWS_S3_REGION_NAME=eu-central-1
AWS_XYFILES_BUCKET_NAME=some-bucket-name
By defining a custom_domain in this manner, we can append the bucket name, something that django-storages doesn't do by default. This ensures Django can access the LocalStack bucket while also returning the correct localhost URL.
I hope this solution works for you!