I am running remote celery worker which takes around 2 hours to complete one task. Broker I have used is rabbitmq and for backend it is rpc. After completion of task, the celery worker doesn't return result back to client.
Celery version is 4.4 and OS is windows
Can someone let me know what could be the issue as I don't see any error in celery logs.
pipenv run celery worker -A src.celery_app -l info -P solo
-------------- celery@remote_host v4.4.7 (cliffs)
--- ***** -----
-- ******* ---- Windows-10-10.0.14393-SP0 2020-09-10 20:00:18
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: celery_app:0x2191f95bb38
- ** ---------- .> transport: amqp://test:**@10.1.9.159:5672/test_host
- ** ---------- .> results: rpc://
- *** --- * --- .> concurrency: 64 (solo)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. doors.routes.abortDoors
. doors.routes.pushDemToDoors
[2020-09-10 20:00:18,572: INFO/MainProcess] Connected to amqp://test:**@10.1.9.159:5672/test_host
[2020-09-10 20:00:18,595: INFO/MainProcess] mingle: searching for neighbors
[2020-09-10 20:00:19,641: INFO/MainProcess] mingle: all alone
[2020-09-10 20:00:19,658: INFO/MainProcess] celery@remote_host ready.
[2020-09-10 20:00:48,533: INFO/MainProcess] Received task: doors.routes.pushDemToDoors[d175239e-dcfe-4282-8596-7be80cf725fe]
Error logs:-
Task doors.routes.pushDemToDoors[697b9f66-d67e-41c0-972f-ca845212a1fb] succeeded in 9341.906000000075s:
[2020-09-12 01:13:01,721: CRITICAL/MainProcess] Couldn't ack 1, reason:ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)
Traceback (most recent call last):
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\kombu\message.py", line 131, in ack_log_error
self.ack(multiple=multiple)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\kombu\message.py", line 126, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\amqp\channel.py", line 1394, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\amqp\abstract_channel.py", line 59, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\amqp\method_framing.py", line 189, in write_frame
write(view[:offset])
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\amqp\transport.py", line 305, in write
self._write(s)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
I didn't find this issue with option -P gevent in celery worker command. The culprit to connection reset issue in my case is -P solo.