djangodockeramazon-sqsdjango-celerypycurl

ImportError: The curl client requires the pycurl library Docker + Django


I'm dockerizing Django 2.2 application and using Pipenv for environment management. I want to use SQS as broker with Django celery.

I have installed the pycurl library using Pipenv

[packages]
...
pycurl = "*"

When I run celery locally

pipenv run celery -A qcg worker -l info

It works but when I run using docker image

docker run app:latest celery -A qcg worker -l info

It gives error

ImportError: The curl client requires the pycurl library.

The command in Docerkfile used to install dependencies

RUN set -ex \
    && BUILD_DEPS=" \
    build-essential \
    libpcre3-dev \
    libpq-dev \
    libcurl4-openssl-dev libssl-dev \
    " \
    && apt-get update && apt-get install -y --no-install-recommends $BUILD_DEPS \
    && export PYCURL_SSL_LIBRARY=nss \
    && pipenv install --deploy --system \
    \
    && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $BUILD_DEPS 
    \
    && rm -rf /var/lib/apt/lists/*

Complete error log

[2020-05-25 17:16:22,216: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/http/__init__.py", line 20, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.7/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
    item()
  File "/usr/local/lib/python3.7/site-packages/vine/promises.py", line 170, in __call__
    return self.throw()
  File "/usr/local/lib/python3.7/site-packages/vine/promises.py", line 167, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/usr/local/lib/python3.7/site-packages/kombu/transport/SQS.py", line 390, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File "/usr/local/lib/python3.7/site-packages/kombu/transport/SQS.py", line 406, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/usr/local/lib/python3.7/site-packages/kombu/transport/SQS.py", line 416, in _get_async
    qname, count=count, connection=self.asynsqs(queue=qname),
  File "/usr/local/lib/python3.7/site-packages/kombu/transport/SQS.py", line 566, in asynsqs
    region=self.region
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 27, in __init__
    **kwargs
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/aws/connection.py", line 194, in __init__
    **http_client_params)
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/aws/connection.py", line 151, in __init__
    self._httpclient = http_client or get_client()
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/http/__init__.py", line 22, in get_client
    client = hub._current_http_client = Client(hub, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/http/__init__.py", line 13, in Client
    return CurlClient(hub, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/http/curl.py", line 43, in __init__
    raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.


Solution

  • This issue was with the libcurl4-nss-dev libssl-dev installation.

    In the following installation script

    RUN set -ex \
        && BUILD_DEPS=" \
        build-essential \
        libpcre3-dev \
        libpq-dev \
        libcurl4-openssl-dev libssl-dev \
        " \
        && apt-get update && apt-get install -y --no-install-recommends $BUILD_DEPS \
        && export PYCURL_SSL_LIBRARY=nss \
        && pipenv install --deploy --system \
        \
        && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $BUILD_DEPS 
        \
        && rm -rf /var/lib/apt/lists/* ```
    

    The libcurl4-nss-dev libssl-dev installation is removed after the Pipenv install. So the package library couldn't be located.

    After separating installation of libcurl4-nss-dev libssl-dev works for me.

    RUN apt-get install -y --no-install-recommends libcurl4-nss-dev libssl-dev