[SOLVED] Cassandra Celery python timeout happens on raw query execution using django db connection execute

Cassandra Celery python timeout happens on raw query execution using django db connection execute

My celery is configured for the Cassandra session like this:

def cassandra_init(*args, **kwargs):
    """ Initialize a clean Cassandra connection. """
    if cql_cluster is not None:
        cql_cluster.shutdown()
    if cql_session is not None:
        cql_session.shutdown()
    connection.setup([settings.DATABASES["default"]["HOST"],], settings.DATABASES["default"]["NAME"])

# Initialize worker context (only standard tasks)
worker_process_init.connect(cassandra_init)

When I am executing a raw cassandra query, timeout happens,

from django.db import connection
cursor = connection.cursor()
total_ap = cursor.execute(
        "SELECT cpu_info  FROM ap_live_stats;")

It works well all over in my django project but not inside the celery tasks.

Error:

[2018-05-09 18:50:21,576: ERROR/ForkPoolWorker-5] Task apps.statistic.tasks.ap_hourly_data_migrator[77a596d4-61a2-43f4-8580-6abc6e9b5866] raised unexpected: OperationTimedOut("errors={'192.168.98.65': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=192.168.98.65",)
Traceback (most recent call last):
  File "/home/vkchlt0079/virtuals/wlc-env/lib/python3.5/site-packages/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/vkchlt0079/virtuals/wlc-env/lib/python3.5/site-packages/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/vkchlt0079/projects/wlcd/src/web_gui/backend/django/wlcd/apps/statistic/tasks.py", line 59, in ap_hourly_data_migrator
    "SELECT cpu_info  FROM ap_live_stats;")
  File "/home/vkchlt0079/virtuals/wlc-env/lib/python3.5/site-packages/django_cassandra_engine/utils.py", line 47, in execute
    return self.cursor.execute(sql)
  File "/home/vkchlt0079/virtuals/wlc-env/lib/python3.5/site-packages/django_cassandra_engine/connection.py", line 12, in execute
    return self.connection.execute(*args, **kwargs)
  File "/home/vkchlt0079/virtuals/wlc-env/lib/python3.5/site-packages/django_cassandra_engine/connection.py", line 86, in execute
    self.session.set_keyspace(self.keyspace)
  File "cassandra/cluster.py", line 2448, in cassandra.cluster.Session.set_keyspace (cassandra/cluster.c:48048)
  File "cassandra/cluster.py", line 2030, in cassandra.cluster.Session.execute (cassandra/cluster.c:38536)
  File "cassandra/cluster.py", line 3844, in cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:80834)
cassandra.OperationTimedOut: errors={'192.168.98.65': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=192.168.98.65

Tried to increase the timeout, but not working and not sure, where it is to be included.

Solution

# project/tasks.py
from celery.signals import worker_init
from django.db import connection

@worker_process_init.connect
def connect_db(**kwargs):
    connection.reconnect()

This will initiate the db connection required via Django Cassandra engine. Reference