python django postgresql zappa python-zappa

ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2._psycopg' zappa

I am using zappa to deploy backend to the AWS Lambda. It worked well, until I decided to use PostgreSQL. I added it in the settings like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': config('DATABASE_NAME'),
        'USER': config('DATABASE_USER'),
        'PASSWORD': config('DATABASE_PASSWORD'),
        'HOST': config('DATABASE_HOST'),
        'PORT': '5432'
    }
}

I am using AWS RDS. I installed psycopg2-binary and also psycopg2 (versions 2.8.6), but the issue remains. The python version is 3.8.

The full error log:

   [1621168086542] [ERROR] ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2._psycopg'
Traceback (most recent call last):
  File "/var/task/handler.py", line 609, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 240, in lambda_handler
    handler = cls()
  File "/var/task/handler.py", line 146, in __init__
    wsgi_app_function = get_django_wsgi(self.settings.DJANGO_SETTINGS)
  File "/var/task/zappa/ext/django_zappa.py", line 20, in get_django_wsgi
    return get_wsgi_application()
  File "/var/task/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/var/task/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/var/task/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/var/task/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/var/task/django/contrib/auth/base_user.py", line 48, in <module>
    class AbstractBaseUser(models.Model):
  File "/var/task/django/db/models/base.py", line 122, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/var/task/django/db/models/base.py", line 326, in add_to_class
    value.contribute_to_class(cls, name)
  File "/var/task/django/db/models/options.py", line 206, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/var/task/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/var/task/django/db/utils.py", line 214, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/var/task/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/var/task/django/db/backends/postgresql/base.py", line 29, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)

Solution

The (likely) issue is that you are building your lambda .zip package on MacOS. When you deploy your lambda function it is running in a Linux environment (specifically, AWS's Linux2 environment). The psycopg2-binary is different for the MacOS vs Linux environments, so if you build your lambda package (including the psycopg2-binary) on a Mac and then deploy to lambda you'll have the issues noted above.

You'll need to build your lambda function inside of an AWS Linux container. Here's a Dockerfile you could use to create a container inside of which you install the psycopg2-binary and build your lambda zip package. Then everything should work:

FROM amazonlinux:2.0.20200207.1

RUN cd /opt && \
    yum install -y gcc openssl-devel bzip2-devel libffi-devel wget tar gzip make && \
    wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz && \
    tar xzf Python-3.8.2.tgz && \
    cd Python-3.8.2 && \
    ./configure --enable-optimizations && \
    make altinstall && \
    rm -f /opt/Python-3.8.2.tgz && \
    echo "alias python3=python3.8" > ~/.bashrc

Note the amaonzonlinux:2.0 operating system, then I just install python 3.8.2 into the environment (you could use a different version of python if desired). From there you can copy in your code and build your lambda .zip package and deploy to lambda.