djangopostgresqlamazon-web-servicesamazon-elastic-beanstalkamazon-rds

Unable to connect to Amazon RDS Postgres Database from my Django Web Application


I am trying to deploy a website using elastic beanstalk and connect it up to a Postgres Database. When trying to makemigrations in django I am getting an error.

I have been able to successfully set up a local Postgres database. I have been trying for a while with no luck.

This is the Error Log

(venv) C:\Users\dania\Development\my_website>python manage.py makemigrations
Traceback (most recent call last):
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\backends\base\base.py", line 217, in ensure_connection
    self.connect()
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\backends\base\base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\backends\postgresql\base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\psycopg2\__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection timed out (0x0000274C/10060)     
        Is the server running on host "my_website.c8lfjtthzko9.ap-southeast-2.rds.amazonaws.com" (13.239.177.165) and accepting
        TCP/IP connections on port 5432?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\core\management\__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\core\management\__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\core\management\base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\core\management\base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\core\management\commands\makemigrations.py", line 101, in handle
    loader.check_consistent_history(connection)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\migrations\loader.py", line 283, in check_consistent_history
    applied = recorder.applied_migrations()
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\migrations\recorder.py", line 73, in applied_migrations
    if self.has_table():
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\migrations\recorder.py", line 56, in has_table
    return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\backends\base\base.py", line 256, in cursor
    return self._cursor()
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\backends\base\base.py", line 233, in _cursor
    self.ensure_connection()
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\backends\base\base.py", line 217, in ensure_connection
    self.connect()
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\backends\base\base.py", line 217, in ensure_connection
    self.connect()
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\backends\base\base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\django\db\backends\postgresql\base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "C:\Users\dania\Development\my_website\venv\lib\site-packages\psycopg2\__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection timed out (0x0000274C/10060)
        Is the server running on host "my_website.c8lfjtthzko9.ap-southeast-2.rds.amazonaws.com" (13.239.177.165) and accepting
        TCP/IP connections on port 5432?

This is the databases section of my project .settings.py file:

from .base import *    
DEBUG = config('DEBUG', cast=bool)
ALLOWED_HOSTS = ['localhost', '127.0.0.1','http://my_website-dev.ap-southeast-2.elasticbeanstalk.com','*******']


AUTH_PASSWORD_VALIDATORS = [
    {'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator'},
    {'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator'},
    {'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator'},
    {'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator'}
]



DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'my_website',
        'USER': '******',
        'PASSWORD': '*********',
        'HOST': 'my_website.*********.ap-southeast-2.rds.amazonaws.com',
        'PORT': '5432'
    }
}

This is my Django Config File

Container_commands:
  01_makemigrations:
    command: "source /opt/python/run/venv/bin/activate && python manage.py makemigrations"
    leader_only: true
  02_migrate:
    command: "source /opt/python/run/venv/bin/activate && python manage.py migrate --noinput"
    leader_only: true
  03_createsu:
    command: "source /opt/python/run/venv/bin/activate && python manage.py createsu"
    leader_only: true
  04_collectstatic:
    command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --noinput"
    leader_only: true

option_settings:
  aws:elasticbeanstalk:application:environment:
    DJANGO_SETTINGS_MODULE: my_website.settings.base
  aws:elasticbeanstalk:container:python:
    WSGIPath: "my_website/wsgi.py"

packages:
    yum:
        httpd24-devel: []

And my requirements.txt

appdirs==1.4.4
autopep8==1.4.4
awsebcli==3.18.2
botocore==1.15.49
cassandra-driver==3.24.0
cement==2.8.2
certifi==2019.3.9
chardet==3.0.4
click==7.1.2
colorama==0.4.3
defusedxml==0.6.0
distlib==0.3.1
Django==2.2
django-allauth==0.39.1
django-countries==5.3.3
django-crispy-forms==1.7.2
django-debug-toolbar==1.10.1
docutils==0.15.2
filelock==3.0.12
future==0.16.0
geomet==0.2.1.post1
idna==2.7
importlib-metadata==1.7.0
importlib-resources==3.0.0
jmespath==0.10.0
mod-wsgi==4.7.1
oauthlib==3.0.1
pathspec==0.5.9
pep8==1.7.1
Pillow==6.2.2
psycopg2-binary==2.8.5
pycodestyle==2.5.0
python-dateutil==2.8.0
python-decouple==3.1
python-slugify==4.0.1
python3-openid==3.1.0
pytz==2018.5
PyYAML==5.3.1
requests==2.20.1
requests-oauthlib==1.2.0
semantic-version==2.5.0
six==1.11.0
slugify==0.0.1
sqlparse==0.2.4
stripe==2.27.0
termcolor==1.1.0
text-unidecode==1.3
urllib3==1.24.2
virtualenv==20.0.30
wcwidth==0.1.9
zipp==3.1.0

I have tried fixing up my Database credentials and creating a new database instance on Amazon RDS but unfortunately no luck.

Thanks for your help in advance!

EDIT: For more information i've added the error log from when I try to deploy this code to elastic beanstalk

Printing Status:
2020-08-16 10:33:18    INFO    createEnvironment is starting.
2020-08-16 10:33:19    INFO    Using elasticbeanstalk-ap-southeast-2-369458984841 as Amazon S3 storage bucket for environment data.
2020-08-16 10:33:40    INFO    Created security group named: sg-049f4a122d881069e
2020-08-16 10:33:43    INFO    Created load balancer named: awseb-e-f-AWSEBLoa-UWLA7I0LF6WN
2020-08-16 10:33:58    INFO    Created security group named: awseb-e-f9zthb6x58-stack-AWSEBSecurityGroup-UXKL3CPFGSVJ
2020-08-16 10:33:58    INFO    Created Auto Scaling launch configuration named: awseb-e-f9zthb6x58-stack-AWSEBAutoScalingLaunchConfiguration-UB2HYII1KHCR
2020-08-16 10:34:46    INFO    Created Auto Scaling group named: awseb-e-f9zthb6x58-stack-AWSEBAutoScalingGroup-1EGDGJYCTFOFQ
2020-08-16 10:34:46    INFO    Waiting for EC2 instances to launch. This may take a few minutes.
2020-08-16 10:35:01    INFO    Created Auto Scaling group policy named: arn:aws:autoscaling:ap-southeast-2:369458984841:scalingPolicy:5681f42e-9672-4f68-8566-fe465044ea90:autoScalingGroupName/awseb-e-f9zthb6x58-stack-AWSEBAutoScalingGroup-1EGDGJYCTFOFQ:policyName/awseb-e-f9zthb6x58-stack-AWSEBAutoScalingScaleUpPolicy-TLZXCZ8CV6Q2
2020-08-16 10:35:01    INFO    Created Auto Scaling group policy named: arn:aws:autoscaling:ap-southeast-2:369458984841:scalingPolicy:13bc6e9d-fd8f-4a12-a2bd-74f036335481:autoScalingGroupName/awseb-e-f9zthb6x58-stack-AWSEBAutoScalingGroup-1EGDGJYCTFOFQ:policyName/awseb-e-f9zthb6x58-stack-AWSEBAutoScalingScaleDownPolicy-JUL92X7VVS7T
2020-08-16 10:35:01    INFO    Created CloudWatch alarm named: awseb-e-f9zthb6x58-stack-AWSEBCloudwatchAlarmHigh-KVUMT41RNBO5
2020-08-16 10:35:01    INFO    Created CloudWatch alarm named: awseb-e-f9zthb6x58-stack-AWSEBCloudwatchAlarmLow-131ZV7XJVRICW
2020-08-16 10:36:03    ERROR   [Instance: i-0bf7de5ced912896c] Command failed on instance. Return code: 1 Output: (TRUNCATED)... in complain
raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.
container_command 02_migrate in .ebextensions/django.config failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2020-08-16 10:36:03    INFO    Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-08-16 10:37:06    ERROR   Create environment operation is complete, but with errors. For more information, see troubleshooting documentation.

And Now included a screenshot of the security group: [1]: https://i.sstatic.net/4LEfH.jpg


Solution

  • Based on the comments and additional info.

    The inbound rule for 5432 port is set to security group sg-597. This allows for inbound traffic only from other instances that have the same sg, not from the internet.

    To enable access to the rds from the internet have to use either 0.0.0.0/0 as the source (i.e. all source locations), or which is better, specific IP range or address (e.g. 1.2.3.4/32). The range can be the range of your work or home public network, and IP can be the specific address of your workstation.