[SOLVED] Run psycopg version 3 on Lambda

Run psycopg version 3 on Lambda

I'm using the lambda docker base image for python3.9

FROM public.ecr.aws/lambda/python:3.9

And I'm trying to use psycopg in my code. Here is a minimum reproducible example:

Dockerfile

# This works!
# FROM python:3.9


# This doesn't work
FROM public.ecr.aws/lambda/python:3.9 

RUN pip install psycopg

COPY . . 

ENTRYPOINT ["python", "script.py"]

script.py

# I'm running these docker commands
# docker build -t test .
# docker rune test

import psycopg

psycopg.connect("host=localhost")

print('hello world')

Using the python3.9 base image I see the correct output. Using the AWS lambda base image I see:

Traceback (most recent call last):
  File "/var/task/script.py", line 4, in <module>
    import psycopg
  File "/var/lang/lib/python3.9/site-packages/psycopg/__init__.py", line 9, in <module>
    from . import pq  # noqa: F401 import early to stabilize side effects
  File "/var/lang/lib/python3.9/site-packages/psycopg/pq/__init__.py", line 114, in <module>
    import_from_libpq()
  File "/var/lang/lib/python3.9/site-packages/psycopg/pq/__init__.py", line 106, in import_from_libpq
    raise ImportError(
ImportError: no pq wrapper available.
Attempts made:
 - couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
 - couldn't import psycopg 'binary' implementation: No module named 'psycopg_binary'
 - couldn't import psycopg 'python' implementation: /lib64/libpq.so.5: undefined symbol: PQconninfo

I've tried installing psycopg-binary instead but my new error there is: Traceback (most recent call last): File "/var/task/script.py", line 4, in <module> import psycopg ModuleNotFoundError: No module named 'psycopg'

I'd like to find a way to use psycopg in a Lambda function. We have older functions using psycopg2 but the lack of maintenance is worrying (psycopg appears more active + supports postgres v14+ and python3.9+) so we'd like to move to psycopg (which now has a version 3 released).

Thanks for your help in advance

Solution

We ditched psycopg entirely and moved to pg8000 which is a pure python implementation and all of the code we wrote was able to port over easily. This way we don't need to worry about binaries or other nonsense.

https://github.com/tlocke/pg8000

Then the simple pip install pg8000 is all our Dockerfile needed and import pg8000.native in our script.py file.