pythonredisceleryworkercloudamqp

What are the consequences of disabling gossip, mingle and heartbeat for celery workers?

What are the implications of disabling gossip, mingle, and heartbeat on my celery workers?

In order to reduce the number of messages sent to CloudAMQP to stay within the free plan, I decided to follow these recommendations. I therefore used the options --without-gossip --without-mingle --without-heartbeat. Since then, I have been using these options by default for all my celery projects but I am not sure if there are any side-effects I am not aware of.

Please note:

Solution

  • This is the base documentation which doesn't give us much info

    heartbeat

    Is related to communication between the worker and the broker (in your case the broker is CloudAMQP). See explanation

    With the --without-heartbeat the worker won't send heartbeat events

    mingle

    It only asks for "logical clocks" and "revoked tasks" from other workers on startup.

    Taken from whatsnew-3.1

    The worker will now attempt to synchronize with other workers in the same cluster.

    Synchronized data currently includes revoked tasks and logical clock.

    This only happens at startup and causes a one second startup delay to collect broadcast responses from other workers.

    You can disable this bootstep using the --without-mingle argument.

    Also see docs

    gossip

    Workers send events to all other workers and this is currently used for "clock synchronization", but it's also possible to write your own handlers on events, such as on_node_join, See docs

    Taken from whatsnew-3.1

    Workers are now passively subscribing to worker related events like heartbeats.

    This means that a worker knows what other workers are doing and can detect if they go offline. Currently this is only used for clock synchronization, but there are many possibilities for future additions and you can write extensions that take advantage of this already.

    Some ideas include consensus protocols, reroute task to best worker (based on resource usage or data locality) or restarting workers when they crash.

    We believe that although this is a small addition, it opens amazing possibilities.

    You can disable this bootstep using the --without-gossip argument.